Feature: modernize API documentation #4607

f18m · 2023-10-23T20:24:08Z

This is a WIP PR.
See mailing list for more details.
See current rendering at: https://f18m.github.io/libzmq/
Current known issues: links in MANPAGES are weird (show some ".html" text)
Additionally this MR requires squashing prior merging.

migrate from the old, unmaintained "asciidoc" tool to the new "asciidoctor" generator; note that Asciidoctor currently works on libzmq docs usin the "compat mode" (see https://docs.asciidoctor.org/asciidoctor/latest/migrate/asciidoc-py/); replacement of section titles with the new Asciidoc syntax is required to get rid of the compat mode.
remove the need of "xmlto" utility to create the manpage output; use asciidoctor for that
add HTML output support to the doc/Makefile by using asciidoctor
change API documentation files extension from .txt to .adoc to make it more explicit that they are Asciidoc-encoded (as a bonus several IDE plugins will autodetect the .adoc format as Asciidoc)
remove asciidoc.conf: asciidoctor does not support that; this also required replacing the macro linkzmq into all documentation pages
add a new Github action CI do deploy the static HTMLs produced by Asciidoctors

f18m · 2023-10-24T09:26:09Z

hi @bluca , @sappo ,
I marked this PR as ready: I did my best to check every possible HTML page and most of the manpages. Everything looks good to me.

Additional steps compared to the PR original description:

converted the asciidoc-py legacy syntax to the "modern" Asciidoc syntax supported by Asciidoctor and other tools
removed references to the "xmlto" and "a2x" tools from the build systems: Asciidoctor can convert the documentation directly to e.g. pdf (via extended converters) and anyway there was no code/target for using "xmlto" and "a2x" tools anyway

Additional tests I've done:

I tested the cmake build on Linux, with/without Asciidoctor installed
I tested the automake build on Linux, with/without Asciidoctor installed
I tested manpage installation

After merging I suggest to go in the github "Settings" and then "Pages" and choose as "source" the "Github Actions" option.
See screenshot taken from my fork:

bluca · 2023-10-24T11:08:38Z

How can this integrate into wikidot where the rest of the documentation is (previous versions, other projects, etc)

sappo · 2023-10-24T11:43:21Z

@bluca it won't. Older docs will still be available in wikidot. This PR will provide the latest docs on gh pages and eventually readthedocs where newer version will then be published to.

bluca · 2023-10-24T11:46:28Z

That's problematic, search engines will always point to the old URLs as they have been around for long, so it would need redirecting, but that can't happen if the previous versions are lost

f18m · 2023-10-24T12:42:36Z

That's problematic, search engines will always point to the old URLs as they have been around for long, so it would need redirecting, but that can't happen if the previous versions are lost

A couple of considerations:

the API documentation published on wikidot can still be updated if there is interest in doing so... I hope it's just a matter of enabling with few more Makefile lines the generation of .xml files out of Asciidoctor. Then there is a Perl script at https://github.com/zeromq/ztools/blob/master/zmqapi/xml2wd.pl in charge of doing the XML->wikidot syntax
My question is: is there still interest? Currently http://api.zeromq.org/ mentions v4.3.2 released back in 2019...
If you agree I think readthedocs.io would be a much better replacement to host libzmq API documentation for multiple versions. After this PR I can volunteer to work on integrating libzmq in readthedocs.io and push there the docs for past versions of libzmq as well. This would become a more modern/usable replacement of the whole http://api.zeromq.org/ website I believe
(there was some email about that also on zeromq-dev mailing list)
I think that if we switch to readthedocs.io most search engines will pick up the change pretty quickly; this might improve current situation: right now the search results for most libzmq API point to very old versions of libzmq... e.g. if I search for "zmq_connect" my Google shows as top result the http://api.zeromq.org/3-2:zmq-connect page, from libzmq 3.2... I'm not sure why it happens... but moving to a newer platform will make latest versions more prominent I guess (just because of SEO of modern websites)

bluca · 2023-10-24T19:38:18Z

I have no opinion on where things are hosted and in which format, as long as we don't break things - if google searches suddenly start showing the wrong website or dead pages, then that's a problem. If it can be made to work with some redirects or aliases or whatnot, then it's all fine by me

f18m · 2023-10-25T08:03:50Z

I have no opinion on where things are hosted and in which format, as long as we don't break things - if google searches suddenly start showing the wrong website or dead pages, then that's a problem. If it can be made to work with some redirects or aliases or whatnot, then it's all fine by me

@sappo do you think it would be possible to setup some redirection from api.zeromq.org to a zeromq.readthedocs.io website (which of course does not exist yet)?
Github pages enabled on libzmq (as suggested in this PR) are a nice way for developers to quickly check their documentation updates but for the zeromq userbase I think it's good to have docs by released version, just like readthedocs.io does, so I keep suggesting to go in that direction.

Looking at https://www.wikidot.com/doc-modules:redirect-module, it seems to be possible to setup such kind of redirect... but I don't have the credentials for the zeromq wikidot website of course...

bluca · 2023-10-25T09:34:33Z

I can do that change too, I have admin access to wikidot as well

f18m · 2023-10-25T09:53:33Z

I can do that change too, I have admin access to wikidot as well

ok wonderful. Then do you think anything else is needed in this PR before merging ?
After this one I can contribute the readthedocs.io integration if that's the agreed solution going forward...

bluca · 2023-10-25T09:56:20Z

The dependencies in packaging/ need to be updated to the new doc build tools before this can be merged, otherwise we'll get no manpages in the packages built on OBS

f18m · 2023-10-25T10:33:04Z

The dependencies in packaging/ need to be updated to the new doc build tools before this can be merged, otherwise we'll get no manpages in the packages built on OBS

done, but I'm not sure how to test it. I found this OBS page:
https://build.opensuse.org/project/monitor/network:messaging:zeromq:ci:zeromq:libzmq:PR-4607

for this PR, but in the "Monitor" tab it shows "Unavailable Build Results."
I could find results only for the "release-stable" package: https://build.opensuse.org/project/monitor/network:messaging:zeromq:release-stable but also there on the row "libzmq" all results are "Broken"...

bluca · 2023-10-25T11:13:25Z

some cache was broken, try to rebase and force push? hopefully it will actually build this time

bluca · 2023-10-25T11:13:42Z

also squash the commit, no need to have it separate

f18m · 2023-10-25T12:27:55Z

done, the branch should be aligned to latest master, all commits squashed... let's see

bluca · 2023-10-25T12:44:13Z

looks like asciidoctor is not available in a bunch of distributions: https://build.opensuse.org/package/show/network:messaging:zeromq:ci:zeromq:libzmq:PR-4607/libzmq

f18m · 2023-10-25T13:02:25Z

looks like asciidoctor is not available in a bunch of distributions: https://build.opensuse.org/package/show/network:messaging:zeromq:ci:zeromq:libzmq:PR-4607/libzmq

I think in some distros (Centos/SuSE) the package is often named "rubygem-asciidoctor"... but that comes typically from EPEL or similar repositories... do you know if e.g. EPEL is enabled or can be enabled during the OBS build?

bluca · 2023-10-25T13:07:02Z

Yes EPEL is enabled: https://build.opensuse.org/projects/network:messaging:zeromq:git-draft/meta
I guess the spec file needs alternative dependencies depending on where it's being built?

f18m · 2023-10-25T13:34:37Z

Yes EPEL is enabled: https://build.opensuse.org/projects/network:messaging:zeromq:git-draft/meta I guess the spec file needs alternative dependencies depending on where it's being built?

I pushed an attempt to install "rubygem-asciidoctor" on RHEL/SuSE and just "asciidoctor" everywhere else. I don't have the possibility to test these many distros myself so I'll watch OBS results and iteratively try to fix all reds... at the end I can re-squash all commits together...

f18m · 2023-10-25T13:49:14Z

Yes EPEL is enabled: https://build.opensuse.org/projects/network:messaging:zeromq:git-draft/meta I guess the spec file needs alternative dependencies depending on where it's being built?

I pushed an attempt to install "rubygem-asciidoctor" on RHEL/SuSE and just "asciidoctor" everywhere else. I don't have the possibility to test these many distros myself so I'll watch OBS results and iteratively try to fix all reds... at the end I can re-squash all commits together...

@bluca do you know if it's possible to enable

https://build.opensuse.org/package/show/devel%3Alanguages%3Aruby%3Aextensions/rubygem-asciidoctor

during the zeromq OBS build?

bluca · 2023-10-25T13:54:58Z

Those are 'devel' projects, IE they are used for development of new packages/versions, and should not be used when doing release builds - so no, however from there it should get included in SUSE releases

bluca · 2023-10-25T13:56:27Z

I think you have to use rubygem(asciidoctor) on suse rather than rubygem-asciidoctor? https://build.opensuse.org/projects/openSUSE:Factory/packages/rubygem-asciidoctor/repositories/standard/binaries/x86_64/ruby3.2-rubygem-asciidoctor-2.0.20-1.1.x86_64.rpm

f18m · 2023-10-25T14:31:56Z

I think you have to use rubygem(asciidoctor) on suse rather than rubygem-asciidoctor? https://build.opensuse.org/projects/openSUSE:Factory/packages/rubygem-asciidoctor/repositories/standard/binaries/x86_64/ruby3.2-rubygem-asciidoctor-2.0.20-1.1.x86_64.rpm

thanks that worked well.
Now only reds are Centos8, RHEL7 and RHEL8.
Centos8 has some trouble installing ruby itself (message is "nothing provides ruby") while RHEL7 and RHEL8 seem to be lacking EPEL (message is "nothing provides rubygem(asciidoctor)").. any chance EPEL can be enabled in RHEL7 and RHEL8?
I googled around about centos8 but so far no luck. Centos8 mentioned by OBS is Centos8 Stream right?

bluca · 2023-10-25T14:33:06Z

yes it's stream - is epel compatible with rhel too? I am not really familiar with the rhel side of the world

f18m · 2023-10-25T14:38:16Z

yes it's stream - is epel compatible with rhel too? I am not really familiar with the rhel side of the world

yes, EPEL is fully compatible with RHEL (and Centos Stream as well btw)... for me it's the opposite: I'm mostly familiar with RHEL side of the world, less with Debian side :)

bluca · 2023-10-25T14:41:00Z

actually it's already configured, both centos and rhel are using the epel repository too

f18m · 2023-10-25T15:02:15Z

actually it's already configured, both centos and rhel are using the epel repository too

then I'm a bit lost... using virtualbox I just run a Centos8 stream, and did successfully:

dnf install -y epel-release  # to get EPEL 
dnf repolist  # this one shows "EPEL8 and EPEL Modular 8" being installed
dnf install -y rubygem-asciidoctor
asciidoctor --version # shows version 2.0.20

not sure if OBS gives some possibility to debug on the Centos8 system? or print e.g. the output of "dnf repolist" somewhere?

bluca · 2023-10-25T15:04:19Z

no that's not possible - maybe rubygem(asciidoctor) is suse specific, and for rhel it needs rubygem-asciidoctor ?

f18m · 2023-10-25T15:34:24Z

no that's not possible - maybe rubygem(asciidoctor) is suse specific, and for rhel it needs rubygem-asciidoctor ?

using both rubygem(asciidoctor) and rubygem-asciidoctor syntax, the error basically does not change.
Do you think it's OK to skip manpage build for Centos8 and RHEL7/RHEL8 OBS builds?
I feel most of RHEL users will take the zeromq RPM coming from EPEL: https://packages.fedoraproject.org/pkgs/zeromq/zeromq/
And my understanding is that the SPEC file used by EPEL is always customized and is not the one in upstream repo... so they have the freedom to enable/disable doc building in EPEL regardless of what we do with OBS.

bluca · 2023-10-25T15:37:40Z

Which package provides ruby(rubygems) on a real installation? We can try a substitution

f18m · 2023-10-25T15:56:50Z

Which package provides ruby(rubygems) on a real installation? We can try a substitution

ruby(rubygems) is provided by the RPM named just rubygems. This is on my vbox Centos8 stream:

Last rpm query with --provides shows that "rubygems" RPM is providing "ruby(rubygems)". I will try the same on RHEL8 but I'm pretty sure it's identical.
Note that "rubygems" RPM comes from the AppStream repo, not even from EPEL...

bluca · 2023-10-25T17:59:22Z

I managed to get it working for centos with a hint from an obs admin, the ruby stuff is in a "module" so it has to be enabled with ExpandFlags: module:ruby-3.0, rhel still doesn't like it tho, will keep trying for a while before giving up

f18m · 2023-10-25T21:11:35Z

I managed to get it working for centos with a hint from an obs admin, the ruby stuff is in a "module" so it has to be enabled with ExpandFlags: module:ruby-3.0, rhel still doesn't like it tho, will keep trying for a while before giving up

oh I see -- nice catch.
This mechanism of "modules" is still something I'm not used to.
Also AFAICT it should have been introduced from RHEL 8 upward. So it's strange to see RHEL7 failing (with EPEL enabled).
But thanks to your efforts I see that now both Centos8 and RHEL8 are green in OBS, very nice.

bluca · 2023-10-25T21:13:22Z

Yeah so you can remove the different name of the dependency, as it seems it's not necessary, and also you can exclude it from rhel/centos 7 or lower so that the build doesn't break, then this can be merged

packaging/redhat/zeromq.spec

f18m · 2023-10-25T21:39:59Z

Yeah so you can remove the different name of the dependency, as it seems it's not necessary, and also you can exclude it from rhel/centos 7 or lower so that the build doesn't break, then this can be merged

done, even if the RHEL7 build in OBS keeps failing, but this time with the same error failure that I see also in libzmq "master" branch: nothing provides python3 needed by

bluca · 2023-10-25T21:41:38Z

yeah you can ignore that, at some point I'll look into it

bluca · 2023-10-25T21:52:44Z

.github/workflows/Docs.yaml

+on:
+  # Runs on pushes targeting the default branch
+  push:
+    branches: ["feature/asciidoctor"]


this needs to be changed to master I guess

also it should filter by repository so that it doesn't run on forks

right, I had mostly forgot about that! It needs to be changed to master.

However I think there is no need to filter by repository... actually I see it as a bonus the fact that forks will deploy github Pages (of course to the forked-repo Pages, e.g. my https://f18m.github.io/libzmq/ site and not to the libzmq official Pages, which will be available at https://zeromq.github.io/libzmq/) since this will allow contributors to check their changes and see how they look like (even if it's also very easy to just produce the HTML locally and preview that locally)...

* migrate from the old, unmaintained "asciidoc-py" tool to the new "asciidoctor" generator * migrate from asciidoc-py syntax to the modern Asciidoc syntax (especially page titles and section titles) * remove the need of "xmlto" utility to create the manpage output; use asciidoctor for that * add HTML output support to the doc/Makefile by using asciidoctor * change API documentation files extension from .txt to .adoc to make it more explicit that they are Asciidoc-encoded (as a bonus several IDE plugins will autodetect the .adoc format as Asciidoc) * remove asciidoc.conf: asciidoctor does not support that; this also required replacing the macro linkzmq into all documentation pages * add a new Github action CI do deploy to Github Pages the static HTMLs produced by Asciidoctors * removed references to the "xmlto" and "a2x" tools from the build and packaging systems: Asciidoctor can convert the documentation directly to e.g. pdf (via extended converters) and anyway there was no code/target for using "xmlto" and "a2x" tools anyway

sappo · 2023-10-25T23:51:32Z

@f18m @bluca great job on this PR. When this new docs have proven themselves we can also introduce them to czmq.

f18m · 2023-10-26T19:55:06Z

@f18m @bluca great job on this PR. When this new docs have proven themselves we can also introduce them to czmq.

Thanks, about czmq: I will save somewhere the script I used to mass convert the asciidoc from legacy to modern format, so we can use that for czmq as well

I'm now looking at readthedocs integration...

When building from dist tarballs, docs come pre-compiled, so no extra dependency is needed. On builds from a git working tree `libzmq` actually used to depend on `asciidoc` and `xmlto` which was recently modernized to `ruby-asciidoctor`, see zeromq/libzmq#4607

f18m requested a review from bluca October 24, 2023 09:20

f18m assigned sappo and bluca Oct 24, 2023

f18m marked this pull request as ready for review October 24, 2023 09:20

f18m mentioned this pull request Oct 24, 2023

api.zeromq.org doesn't use HTTPS zeromq/zeromq.org#131

Closed

f18m force-pushed the feature/asciidoctor branch from e7db1c5 to 20c506a Compare October 24, 2023 10:30

f18m force-pushed the feature/asciidoctor branch from 4edce26 to af79607 Compare October 25, 2023 12:27

f18m force-pushed the feature/asciidoctor branch from 44d4472 to 56cc3c4 Compare October 25, 2023 21:37

bluca reviewed Oct 25, 2023

View reviewed changes

packaging/redhat/zeromq.spec Outdated Show resolved Hide resolved

bluca reviewed Oct 25, 2023

View reviewed changes

f18m force-pushed the feature/asciidoctor branch from 56cc3c4 to b6ca9b2 Compare October 25, 2023 22:00

bluca approved these changes Oct 25, 2023

View reviewed changes

bluca merged commit f0c471a into zeromq:master Oct 25, 2023
46 of 47 checks passed

f18m deleted the feature/asciidoctor branch October 26, 2023 19:51

Feature: modernize API documentation #4607

Feature: modernize API documentation #4607

Conversation

f18m commented Oct 23, 2023 • edited Loading

f18m commented Oct 24, 2023 • edited Loading

bluca commented Oct 24, 2023

sappo commented Oct 24, 2023

bluca commented Oct 24, 2023

f18m commented Oct 24, 2023 • edited Loading

bluca commented Oct 24, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023 • edited Loading

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023 • edited Loading

bluca commented Oct 25, 2023

f18m commented Oct 25, 2023

bluca commented Oct 25, 2023

bluca Oct 25, 2023

Choose a reason for hiding this comment

bluca Oct 25, 2023

Choose a reason for hiding this comment

f18m Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

sappo commented Oct 25, 2023

f18m commented Oct 26, 2023

f18m commented Oct 23, 2023 •

edited

Loading

f18m commented Oct 24, 2023 •

edited

Loading

f18m commented Oct 24, 2023 •

edited

Loading

f18m commented Oct 25, 2023 •

edited

Loading

f18m commented Oct 25, 2023 •

edited

Loading

f18m Oct 25, 2023 •

edited

Loading