Sympa REST?

[ikedas]

I found a branch on Renater/sympa working on REST implementation.

I've not got good look at its details, but will it have anything to do with work on sympa-service-rest ? (may they be eventually merged?)

[dverdin]

Hi Soji, and sorry for not answering this.

This is a contribution we plan to do to the Sympa community - if the community agrees with it. several of our users were asking us for it.

We have reached an already satisfactory point: The REST interface we developed has the same set of functionalities as the SOAP interface, plus message sending, moderation, etc.

If you want a full set of functionalities, you can read the openAPI file. It is up to date with the code.

All the code is base on the Sympa Spindle and Request handlers.

We implemented full tests on the new code. In order to be able to run them with a simple prove, we located these tests in a t/ sub-directory.

In addition to the REST interface, we created a Sympa::API.pm module which serves as an interface between Sympa modules and the REST.pm module, that handles HTTP queries and answers. We feel it could be used to remove duplicated code such as in SOAP, for example.

We would really like merge our code into the Sympa community project. I'd be very glad we could start a discussion about it.

Cheers,

David

[racke]

A REST interface is indeed a long awaited feature for Sympa. But ... this will be actually hard to review and merge. A better way would be to submit PRs for the code changing in existing modules. And I'm thinking off small changes per each PR.

[dverdin]

Not to be loud but you should have a look at what we did. It is mostly orthogonal to the current codebase. We did our best not to disturb anything else.

This is mostly a new interface, based on the current Sympa modules. So it is very easy to merge.

Everything is located in src/lib/Sympa/API.

[racke]

In that case the branch should be rebased on current sympa-6.2.

[dverdin]

Yes. We did not keep it in sync that last months (health troubles on my part, sorry).

Actually, if you leave us a few days, we could make a small presentation about what we did to you and a few other - assuming you have the time, of course. A web conference could help us explain what we did way better and it would certainly ease the review process.

Just let me know.

Cheers!

David

[ikedas]

I looked Renater/etienne-debut at glance:

• Diff against 6.2.67b.1

My suggestions:

It should be devided into three pieces.

1. Obvious fixes of bugs and so on in the main project (sympa-community/sympa)
2. Additions or extensions of Sympa's internal API (Sympa::Request::Hander modules) or the other functionalities added into the main project
3. Additions specific to REST API (modules under Sympa::API, related entries in cpanfile and tests)

Each piece should be treated as follows.

• Items 1. should be submitted on main project as issues about the bugs with attached PR if any (anyways submitting issue is recommended).
• Items 2. should be submitted on main project as issue(s) about enhancements.
• Items 3. should be maintained as the separate project. If possible, I think sympa-service-rest would be better to take over them.

---

The following are not suggestions, but just my own miscellaneous impressions.

• The name API looks too generic for me.I feel Sympa::REST is more fitting.
• On the separate project above: I think the current autotools layout for Sympa project (where almost everything is under src/) is a bit too complex, and REST API projects could adopt the CPAN layout (where lib, man, script, t, etc. are directly under the top).
• As a reminder: I'm planning to make some changes on set and (possiblly) move_user request handlers in these some months. I think it will be the last of the refactoring work I have been continuing over the last few years. Keep it mind if you plan to use thsese internal API.

Another important thing: @yent would be better to be added to our team.

[racke]

@ikedas Valid points 👍
@dverdin Yes, we can make a video meeting.

[salaun-urennes1]

Great job, David !
@Soji, I don't understand why you consider (3) should be outside of the main Sympa project?

[racke]

@salaun-urennes1 It could live on CPAN and it shouldn't need to share the complex layout of the Sympa project. Also it is optional.

@ikedas 4. Deprecate SOAP API in favour of the REST API.

[salaun-urennes1]

@racke Of course the REST API could be maintained as a separate CPAN module, but a REST API sounds like a core module. With a reliable and efficient REST service, most of Sympa daemons could be redesigned to become REST clients.

[ikedas]

@Soji, I don't understand why you consider (3) should be outside of the main Sympa project?

Simply because its codebase need not have a duplicate of the Sympa codebase.

[EDIT] Each of the interfaces (currently mail, web and SOAP) would be better to be optional. In fact, currently Sympa works even if any of these are disabled (e.g. without configuring sympa address). The reason they live together in one code base is simply because refactoring to separate them has taken time. I see no reason to mix REST in it.

[ikedas]

@ikedas 4. Deprecate SOAP API in favour of the REST API.

I agree. That's also what we have waited for!

[dverdin]

Thanks for all these observations.

Let me answer them following the incoming order:

Obvious fixes of bugs and so on in the main project (sympa-community/sympa)

Indeed. We fixed a few problems along the way and these should be separated from the main REST contribution.

Additions or extensions of Sympa's internal API (Sympa::Request::Hander modules) or the other functionalities added into the main project

Certainly. However, they are used by our REST interface only (for now). So these new modules will look weird in the current code base.

Additions specific to REST API (modules under Sympa::API, related entries in cpanfile and tests)

Yep!

Each piece should be treated as follows.

Items 1. should be submitted on main project as issues about the bugs with attached PR if any (anyways submitting issue is recommended).

Yep.

Items 2. should be submitted on main project as issue(s) about enhancements.

Yep.

Items 3. should be maintained as the separate project. If possible, I think sympa-service-rest would be better to take over them.

I don't think so. The Sympa REST service can certainly be shipped separately, but:
1- it is useless without Sympa
2- Some users would appreciate to have it shipped directly with Sympa, without having to install something separately.
3- We always shipped all Sympa components together. That would be a change in our shipping policy. I remember that this issue was tackled during the 20 years hackathon. Some people wanted to split Sympa between its different interfaces and ship them separately. It is certainly possible, especially with all the refactoring Soji did during the last years. But I am not sure our user base is large enough to justify the effort.
This move would have its virtues: preventing code duplications in the future, for example.

The following are not suggestions, but just my own miscellaneous impressions.

The name API looks too generic for me.I feel Sympa::REST is more fitting.

Yes, we had a discussion about it. The current setup is mainly due to our will to make a least intrusive possible contribution, and so we put everything in Sympa::API, which can make sense if you consider putting here the libraries used to implement an interface. But that is a different issue.

On the separate project above: I think the current autotools layout for Sympa project (where almost everything is under src/) is a bit too complex, and REST API projects could adopt the CPAN layout (where lib, man, script, t, etc. are directly under the top).

If decided to make it a different project, why not. But I don't think we should.

As a reminder: I'm planning to make some changes on set and (possiblly) move_user request handlers in these some months. I think it will be the last of the refactoring work I have been continuing over the last few years. Keep it mind if you plan to thses internal API.

Thanks a lot for the input! We will certainly be careful about it. We make tests for each REST function anyway, so we would notice the change.

Another important thing: @yent would be better to be added to our team.

Oh, certainly. He is Étienne Méléard, long-time listmaster and Sympa developer here at RENATER.

@dverdin Yes, we can make a video meeting.

Cool! Next week? Who would like to participate?
I don't plan to show the whole code, just how it works and how it is articulated with the Sympaa code base.

@salaun-urennes1 It could live on CPAN and it shouldn't need to share the complex layout of the Sympa project. Also it is optional.
I understand the interest. However, I don't see how it sould be used by anything else than Sympa

Great job, David !
@Soji, I don't understand why you consider (3) should be outside of the main Sympa project?

Thanks!

[EDIT] Each of the interfaces (currently mail, web and SOAP) would be better to be optional. In fact, currently Sympa works even if any of these are disabled (e.g. without configuring sympa address). The reason they live together in one code base is simply because refactoring to separate them has taken time. I see no reason to mix REST in it.

The reason for distributing REST with the code base would be to have all the needed code in one installation.

On this topic: In a nutshell, we have the following architecture:

• An executable file call Sympa::API::REST->to_app.
• Sympa::API::REST is only capable of interpreting HTTP requests and sending answers.
• In order to actually handle the query, it calls a sub in Sympa::API, which in turns call the correct Request Handlers.

We think that, in the future:
1- we could use Sympa::API in all interfaces : web, mail, CLI, REST and SOAP (for the time it will last). Refactoring these executables to use Sympa::API would remove duplicate code.
2- Based on the REST API, we could have a new web interface. That's a tricky job because of the different ways of customization the current interface offers, thanks to Template::Toolkit.

Sorry for this long post, I hope it answers the different remarks and suggestions. If by any means I have been unclear, please let me know.

[ikedas]

@dverdin, your plan in the future is interesting. However, that would be quite some time in the future, if ever, to happen (The code has not even been reviewed yet). Developers of REST should concentrate on their own work for the time being. Even for this reason alone, I think there is no reason to align the code base and release cycle with the main project.

Also, it is more difficult to separate something once it has been mixed than vice versa. Why not start with a separate project before discussing this point any further?

[dverdin]

@ikedas That's a pretty valid point indeed. And actually separating development project does not mean that we won't ship them together. This is mainly a way of keeping things separated while developing.

I see one difficulty: We still need Sympa modules to run the REST API and, from a developer point of view, run our unit tests. When we add a user to a list from the REST interface, we still need the add.pm request handler which is in the . Right now, I don't see how to do this simply. I might be missing something on git projects articulation.
Could someone help me about this?

[ikedas]

@racke, would you mind if I merge the work of @yent into sympa-service-rest? I think it would be desirable for us to work together.

(Incidentally, somebodies seem to have forgotten that this is one of the projects proposed in the 2018 "Sympa Hackathon". Also by this reason I feel it natural that this project will continue working.)

---

Related note: It would be nice to think about working with sympa-vue. This was also proposed earlier and has been maintained by @racke).

However, the project faces some challenges.

• Code needs to be restructured. A simple update is not sufficient, because it already contains deprecated packages (I've tried to solve this by editing only package.json but could not do it with my knowledge).
• There are several other suggestions for the UI, but they have not been incorporated into this project so far. For example New data sources component #20 suggested by @sandhose (Data sources component sympa-vue#1 assigned to @ludovicm67 ).

[dverdin]

EDIT: Rephrasing my post which had a very poor choice of words.

@ikedas It is a bad idea to simply merge the two projects. They have not the same structure, it will be a mess and nothing good will come out of it.

There may be good ideas to take from both but a simple merge will be counter productive.

As a side note, I'd like to make it clear that, though the code was made by @yent and me, we both did it on our time working for RENATER.

[dverdin]

I'll post a survey to find a time slot to present our work.
For code survey purpose, it will be probably necessary to squash commits. I'll try to find time to do it.

[ikedas]

I am sorry, David, but would you please shut up?

You have often said "let us do as we please" and even said "you stop working". But you show little interest in the practical running of the community. I don't trust such a person who repeatedly make childish demands.

To begin with, your words have often disappointed me over the past years (and when I explained it, you have repeated your excuses). I prefer to talk with persons who are sincere in what they do. I don't want to talk to someone who is almost talk without do.

Please don't waste any more time of ours with your empty words.

---

I want to make it clear to everyone here. We have no deal. If you are here for any other reason than to help the community, please leave.

[ikedas]

In other words, if you want to manipulate the community the way the particular organization, RENATER wants to, please refrain from joining the community.

For example, I am allowed by my company that I work for the Sympa community to use some of my work time. They also provide us with some equipment (such as a server). However, my opinions in the community are personal. I and my company confirm that the company does not intervene in the running of the Sympa community.

[racke]

@dverdin Please keep in mind that @ikedas is the brain and heart of Sympa. Of course the community made many contributions, but the lion's share (majority) was done by him.

However, merging work of @yent into sympa-service-rest doesn't make sense to me. The code in this repo was just the beginning and was never continued.

[salaun-urennes1]

I am sorry, David, but would you please shut up?

Hum...does it really help?

[ikedas]

Edit after post: Some typos and additions

---

I am sorry, David, but would you please shut up?

Hum...does it really help?

We have no deal. If your ultimate goal is to make us swallow RENATER's opinion, you don't even have to be here, do you?

BTW have you read the GitHub Docs?

We are no longer developing only with like-minded small group like cru. So the way we work on projects is different. We no longer use centralised code-sharing systems like CVS or Subversion, but distributed collaboration based on forking and pull requests and/or multiple divided projects.

It seems to me that David doesn't understand this and is only insisting on "giving us total control", "let us do what we please".

Olivier, could you please learn about GitHub and what we have been doing for this several years?

[dverdin]

Hey dudes,
(edit : typo)
Reposting, it looks like my answer vanished somehow.

I did not understand everything Soji said. At least I do not recognize me in the description he did of me, but I can't change his feelings. I know how hard it can be to be the sole person on which the project stands, so I understand his resentment.

Moving on: I am here because Sympa is a tool, and RENATER needs this tool improved with a REST interface. So Étienne and I developed it. Period.

I have no interest in running the community, neither does RENATER. I am pretty happy that somebody else is doing it.

Is it clearer that way?

[ikedas]

I have no interest in running the community, neither does RENATER. I am pretty happy that somebody else is doing it.

As I wrote, @dverdin, if you are not interested in running the community, please feel free to leave our team: Even if doing so, you can still contribute to the community.

@yent, if you (and RENATER) are still willing to contribute your work, I'll begin investigating it in this weekend. (Note: Of course, your name and affiliation will be noted on the documentation of the code).

[dverdin]

I did not understand that the "everyone" team contained only people willing to run the community.

If it is the case, from what I read in the issues, there should probably not be 20 people in this team. I know for sure that Étienne has no intention of running anything. He just wants to contribute code.

Can you confirm it ? The team is intended for people willing to run the community?

If that is the case, of course I will leave it.

[dverdin]

(But just before I leave)

A survey to find a time slot for us to present the REST interface : https://evento.renater.fr/survey/sympa-rest-interface-presentation-pkfd4sr5

Thanks!

[racke]

As the name implies, everyone are all members of the Github community. This is the only team we have right now.
The people actually "running" the community would be a smaller circle.

[ikedas]

@racke,

However, merging work of @yent into sympa-service-rest doesn't make sense to me. The code in this repo was just the beginning and was never continued.

Indeed. If we merged, @yent's work would mostly override the base you've prepared. That is, the merge would only have symbolic significance: Working together.

Anyway, I'll delete sympa-service-rest repo (or you may delete it) and create a new repository --- "sympa-rest"?

[ikedas]

The project was abandoned. This discussion is closed.