[Tiki-devel] Composer Dependencies Revamp

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[Tiki-devel] Composer Dependencies Revamp

Ricardo Melo
Hi all,

I've been working on a propose regarding the way composer is used today in Tiki and if we can use composer / some kind of packaging to address a couple of use cases that we face today.


And the original considerations / goals around this topic are in : https://dev.tiki.org/Install+optional+libraries+in+Tiki+via+Composer

From them there are a couple of different use cases I would like to highlight, that I think this Revamp can help address:

* If people add (locally) a library to a tiki installation avoid merge conflicts
* Allow people to install packages, that because of the licence, can't be bundled with tiki (mPDF is a good example)
* Avoid bundle with Tiki HUGE packages that might not be used/activated in most tiki installations (and still support easy installation of the packages to those without shell access to server - only FTP or similar)

There is wide range of different approaches based on composer, I've highlighted not only my suggestion but also some other options that are used by other projects but may not be ideal for tiki.

I would deeply appreciate that you take a couple of minutes to review the document, feel free to edit if anything don't seem ok and/or raise questions here on the mailing list (I'll do my best to answer them as soon as possible).

The goal is to implement the outcome of this discussion for Tiki 17, so I really appreciate your input on this topic.

Thank you very much,
Ricardo

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Composer Dependencies Revamp

Dr. Sassafras
Entertain me for a minute. My thoughts about this are evolving.

What if we made ALL libraries treated the same way: upgradable/installable/deletable. Libraries being used (based on enabled preferences/core functionality) can be blocked from uninstalling. Some of these libraries we can bundle and have default options as installed, but in essence it they be treated exactly the same.

I would also love to see a feature where minor releases can be automatically/user initiated upgraded for security and fixes.

A lot of intranets with no internet access wont be able to download packages. So what if we include a folder with all the bundled libraries, so they are not installed, but are there..... It wont save bandwidth, but it will:
*Add ease of installing libs with licenses that are incompatible
*Allow fast security upgrades
*Prevent security issues in libs that are not being used
*Make managing tiki via FTP easier (more on that in a bit)

In such a case, tiki could always, by default install the included zip, rather than checking for a download. This would mean minimal fussing when features are turned on. It would almost always ensure that minimal fussing is needed when turning on features. If there is an update for the lib, it could be handled ether through an auto/user-initeated process.

How will that work with upgrades.... while Im guessing we may need a separate list packaged with each update, that has the minimum requirements of each vendor, there will be major release updates here. Part of the upgrade-install process could be to check for vendor version requirements and install the packaged lib if its needed. Any libraries needed for core functionality should of course not be included in zip, in favor of being already installed.

Coming back to the FTP. Its impossible to upload a deflated version of tiki with ftp. At lest not on any connection ive ever used. The issue is the number of files, rather than the size. If we can reduce the vendors to zips, so its only tiki files left, it might just be doable, or at least a lot closer. Deleting tiki through a FTP program will also WAY faster.

Including the files will also make things like tiki show instances much easier. Im guessing there may be a lot of errors if the files needed to be downloaded, and the user does not have access to FTP in those circumstances.

Including the zips would also prevent the issue of a lib not being available when its needed. What if a server does down (even for 10 minutes) It could be a real hassle.

We could also (if its needed) to have a revert button to revert a upgraded back to the one included originally with tiki.

I am also guessing that including the libs might make tiki profiles work better, where there can be a lot of preferences turned on at once.

Im guessing though that it would be posable to release a version of tiki without the bundled packages, but im really not convinced that everything will work well without fussing over the vendors control panel, and possibly manual downloads and FTP to enable tiki features.... or even failed attempts in some cases. Not all admins will be able to FTP/Shell

I do have a question as to how this will effect the "Tiki Security Check"... I have not put much thought into it, but its likely major changes will need to take place there.

That is as far as my thoughts have gone on this.

On Mon, Jan 16, 2017 at 7:08 AM, Ricardo Melo <[hidden email]> wrote:
Hi all,

I've been working on a propose regarding the way composer is used today in Tiki and if we can use composer / some kind of packaging to address a couple of use cases that we face today.


And the original considerations / goals around this topic are in : https://dev.tiki.org/Install+optional+libraries+in+Tiki+via+Composer

From them there are a couple of different use cases I would like to highlight, that I think this Revamp can help address:

* If people add (locally) a library to a tiki installation avoid merge conflicts
* Allow people to install packages, that because of the licence, can't be bundled with tiki (mPDF is a good example)
* Avoid bundle with Tiki HUGE packages that might not be used/activated in most tiki installations (and still support easy installation of the packages to those without shell access to server - only FTP or similar)

There is wide range of different approaches based on composer, I've highlighted not only my suggestion but also some other options that are used by other projects but may not be ideal for tiki.

I would deeply appreciate that you take a couple of minutes to review the document, feel free to edit if anything don't seem ok and/or raise questions here on the mailing list (I'll do my best to answer them as soon as possible).

The goal is to implement the outcome of this discussion for Tiki 17, so I really appreciate your input on this topic.

Thank you very much,
Ricardo

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Composer Dependencies Revamp

Ricardo Melo
Brendan,

Thank you very much for your feedback.

Just to make sure that I understood the vision:

- Tiki would be the tiki code plush the minimal set of vendors to be able to run tiki.
- All other libraries that are needed based on the features that the user enable, should be distributed as a big zip (or as individual zips) with all extra dependencies
- I think libraries with licences incompatibles with tiki may not be distributed even as zip, as part of the distribution

Then when you enable a feature (that needs to declare the dependencies), tiki will look into the zip and extract the corresponding library.

(I'll leave the part about updates / auto updates aside to focus only in the library distribution)

Does this describes your ideia?

Thank you
Ricardo


On Mon, Jan 16, 2017 at 12:55 AM, Brendan Ferguson <[hidden email]> wrote:
Entertain me for a minute. My thoughts about this are evolving.

What if we made ALL libraries treated the same way: upgradable/installable/deletable. Libraries being used (based on enabled preferences/core functionality) can be blocked from uninstalling. Some of these libraries we can bundle and have default options as installed, but in essence it they be treated exactly the same.

I would also love to see a feature where minor releases can be automatically/user initiated upgraded for security and fixes.

A lot of intranets with no internet access wont be able to download packages. So what if we include a folder with all the bundled libraries, so they are not installed, but are there..... It wont save bandwidth, but it will:
*Add ease of installing libs with licenses that are incompatible
*Allow fast security upgrades
*Prevent security issues in libs that are not being used
*Make managing tiki via FTP easier (more on that in a bit)

In such a case, tiki could always, by default install the included zip, rather than checking for a download. This would mean minimal fussing when features are turned on. It would almost always ensure that minimal fussing is needed when turning on features. If there is an update for the lib, it could be handled ether through an auto/user-initeated process.

How will that work with upgrades.... while Im guessing we may need a separate list packaged with each update, that has the minimum requirements of each vendor, there will be major release updates here. Part of the upgrade-install process could be to check for vendor version requirements and install the packaged lib if its needed. Any libraries needed for core functionality should of course not be included in zip, in favor of being already installed.

Coming back to the FTP. Its impossible to upload a deflated version of tiki with ftp. At lest not on any connection ive ever used. The issue is the number of files, rather than the size. If we can reduce the vendors to zips, so its only tiki files left, it might just be doable, or at least a lot closer. Deleting tiki through a FTP program will also WAY faster.

Including the files will also make things like tiki show instances much easier. Im guessing there may be a lot of errors if the files needed to be downloaded, and the user does not have access to FTP in those circumstances.

Including the zips would also prevent the issue of a lib not being available when its needed. What if a server does down (even for 10 minutes) It could be a real hassle.

We could also (if its needed) to have a revert button to revert a upgraded back to the one included originally with tiki.

I am also guessing that including the libs might make tiki profiles work better, where there can be a lot of preferences turned on at once.

Im guessing though that it would be posable to release a version of tiki without the bundled packages, but im really not convinced that everything will work well without fussing over the vendors control panel, and possibly manual downloads and FTP to enable tiki features.... or even failed attempts in some cases. Not all admins will be able to FTP/Shell

I do have a question as to how this will effect the "Tiki Security Check"... I have not put much thought into it, but its likely major changes will need to take place there.

That is as far as my thoughts have gone on this.

On Mon, Jan 16, 2017 at 7:08 AM, Ricardo Melo <[hidden email]> wrote:
Hi all,

I've been working on a propose regarding the way composer is used today in Tiki and if we can use composer / some kind of packaging to address a couple of use cases that we face today.


And the original considerations / goals around this topic are in : https://dev.tiki.org/Install+optional+libraries+in+Tiki+via+Composer

From them there are a couple of different use cases I would like to highlight, that I think this Revamp can help address:

* If people add (locally) a library to a tiki installation avoid merge conflicts
* Allow people to install packages, that because of the licence, can't be bundled with tiki (mPDF is a good example)
* Avoid bundle with Tiki HUGE packages that might not be used/activated in most tiki installations (and still support easy installation of the packages to those without shell access to server - only FTP or similar)

There is wide range of different approaches based on composer, I've highlighted not only my suggestion but also some other options that are used by other projects but may not be ideal for tiki.

I would deeply appreciate that you take a couple of minutes to review the document, feel free to edit if anything don't seem ok and/or raise questions here on the mailing list (I'll do my best to answer them as soon as possible).

The goal is to implement the outcome of this discussion for Tiki 17, so I really appreciate your input on this topic.

Thank you very much,
Ricardo

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Composer Dependencies Revamp

Dr. Sassafras
Yes. That is it.

I think it makes sense to have them all packaged in separate zips. Its already the way composer works, running md5's would be easier and there might be performance issues zipping and unzipping all the vendors combined.

The only addition I would make to this is that these libraries are not required. If a feature is enabled that needs one of these libraries, it first looks for the local zip, and then falls back to downloading the dependency.

I guess the first step would be to implement this without the complication of downloads. Being able to depend on the zip always being there simplifies the code somewhat, and may be easier to debug the core code without it.

One could start with all the libraries that composer currently uses, and leave vendors_extra to be treated as part of the tiki code.

I think the tricky bit might be sorting through all the features-preferences and figuring out what libraries they need. Im imagination an extra dependencies setting in the configuration settings for preferences, something similar with features.

Brendan

On Fri, Jan 27, 2017 at 7:09 AM, Ricardo Melo <[hidden email]> wrote:
Brendan,

Thank you very much for your feedback.

Just to make sure that I understood the vision:

- Tiki would be the tiki code plush the minimal set of vendors to be able to run tiki.
- All other libraries that are needed based on the features that the user enable, should be distributed as a big zip (or as individual zips) with all extra dependencies
- I think libraries with licences incompatibles with tiki may not be distributed even as zip, as part of the distribution

Then when you enable a feature (that needs to declare the dependencies), tiki will look into the zip and extract the corresponding library.

(I'll leave the part about updates / auto updates aside to focus only in the library distribution)

Does this describes your ideia?

Thank you
Ricardo


On Mon, Jan 16, 2017 at 12:55 AM, Brendan Ferguson <[hidden email]> wrote:
Entertain me for a minute. My thoughts about this are evolving.

What if we made ALL libraries treated the same way: upgradable/installable/deletable. Libraries being used (based on enabled preferences/core functionality) can be blocked from uninstalling. Some of these libraries we can bundle and have default options as installed, but in essence it they be treated exactly the same.

I would also love to see a feature where minor releases can be automatically/user initiated upgraded for security and fixes.

A lot of intranets with no internet access wont be able to download packages. So what if we include a folder with all the bundled libraries, so they are not installed, but are there..... It wont save bandwidth, but it will:
*Add ease of installing libs with licenses that are incompatible
*Allow fast security upgrades
*Prevent security issues in libs that are not being used
*Make managing tiki via FTP easier (more on that in a bit)

In such a case, tiki could always, by default install the included zip, rather than checking for a download. This would mean minimal fussing when features are turned on. It would almost always ensure that minimal fussing is needed when turning on features. If there is an update for the lib, it could be handled ether through an auto/user-initeated process.

How will that work with upgrades.... while Im guessing we may need a separate list packaged with each update, that has the minimum requirements of each vendor, there will be major release updates here. Part of the upgrade-install process could be to check for vendor version requirements and install the packaged lib if its needed. Any libraries needed for core functionality should of course not be included in zip, in favor of being already installed.

Coming back to the FTP. Its impossible to upload a deflated version of tiki with ftp. At lest not on any connection ive ever used. The issue is the number of files, rather than the size. If we can reduce the vendors to zips, so its only tiki files left, it might just be doable, or at least a lot closer. Deleting tiki through a FTP program will also WAY faster.

Including the files will also make things like tiki show instances much easier. Im guessing there may be a lot of errors if the files needed to be downloaded, and the user does not have access to FTP in those circumstances.

Including the zips would also prevent the issue of a lib not being available when its needed. What if a server does down (even for 10 minutes) It could be a real hassle.

We could also (if its needed) to have a revert button to revert a upgraded back to the one included originally with tiki.

I am also guessing that including the libs might make tiki profiles work better, where there can be a lot of preferences turned on at once.

Im guessing though that it would be posable to release a version of tiki without the bundled packages, but im really not convinced that everything will work well without fussing over the vendors control panel, and possibly manual downloads and FTP to enable tiki features.... or even failed attempts in some cases. Not all admins will be able to FTP/Shell

I do have a question as to how this will effect the "Tiki Security Check"... I have not put much thought into it, but its likely major changes will need to take place there.

That is as far as my thoughts have gone on this.

On Mon, Jan 16, 2017 at 7:08 AM, Ricardo Melo <[hidden email]> wrote:
Hi all,

I've been working on a propose regarding the way composer is used today in Tiki and if we can use composer / some kind of packaging to address a couple of use cases that we face today.


And the original considerations / goals around this topic are in : https://dev.tiki.org/Install+optional+libraries+in+Tiki+via+Composer

From them there are a couple of different use cases I would like to highlight, that I think this Revamp can help address:

* If people add (locally) a library to a tiki installation avoid merge conflicts
* Allow people to install packages, that because of the licence, can't be bundled with tiki (mPDF is a good example)
* Avoid bundle with Tiki HUGE packages that might not be used/activated in most tiki installations (and still support easy installation of the packages to those without shell access to server - only FTP or similar)

There is wide range of different approaches based on composer, I've highlighted not only my suggestion but also some other options that are used by other projects but may not be ideal for tiki.

I would deeply appreciate that you take a couple of minutes to review the document, feel free to edit if anything don't seem ok and/or raise questions here on the mailing list (I'll do my best to answer them as soon as possible).

The goal is to implement the outcome of this discussion for Tiki 17, so I really appreciate your input on this topic.

Thank you very much,
Ricardo

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Composer Dependencies Revamp

Marc Laporte-3
In reply to this post by Dr. Sassafras
Hi Brendan,

Thank you for your thoughts.

I will follow up on a few points:


"It's impossible to upload a deflated version of Tiki with FTP."
---------------------------------------------------------------------------------------
It is a problem indeed. Sometimes, the upload completes (or so you
think), but some files are missing and it creates all kinds of
problems. And since it's random files, the resulting "bug" is
different than anybody else in the community gets, so no one knows how
to help. Good thing we have
https://dev.tiki.org/show.tiki.org+Overview

Here is a way to detect missing, corrupted or hacked files:
https://doc.tiki.org/Security+Admin#Check_your_files
This integrity check is great but keep in mind that a more
sophisticated attacker could modify the signatures to remain
undetected.

TRIM's "make check" permits an integrity check from a remote server
(and thus, signatures are safe) and it takes into account any
intentionally modified files: https://doc.tiki.org/TRIM#make_check



Back on the idea of reducing large number of small files: perhaps a
solution is to look into PHARs:
http://php.net/manual/en/intro.phar.php

1- For each external library
2- Perhaps for the whole vendor directory
3- Perhaps for Tiki libs
4- And why not for Tiki itself?

I didn't compare the options, but there are many tools to manage PHARs such as
https://box-project.github.io/box2/
https://phar.io/
https://github.com/clue/phar-composer


I am not so concerned by disk space but I am concerned that some files
can be active even when a feature is de-activated. For Tiki files, we
have a good process:
https://dev.tiki.org/How+to+release#Check_that_all_PHP_files_have_a_feature_check

Some of the recent vulnerabilities in Tiki were caused by the presence
of external libs, and thus, without a feature check.
https://www.exploit-db.com/exploits/40080/
https://www.exploit-db.com/exploits/40091/

If libraries are PHAR-ed, perhaps we could add a similar protection?
This could perhaps enhance or replace what you added here:
https://sourceforge.net/p/tikiwiki/code/HEAD/tree/trunk/vendor/.htaccess

CMS Made Simple uses PHARs "A Phar is a single, self contained,
executable PHP Archive. It allows us to distribute the CMSMS
installation assistant as a single file even though it contains
numerous libraries, classes, stylesheets, and scripts. This allows
CMSMS users to install, upgrade or freshen their CMSMS systems by
uploading a single file to their web server."
Source: https://docs.cmsmadesimple.org/installation/installing

I think Tiki would need some work to get there. Notably:
A single directory (with sub-directories) for all temp and cache content
A single directory (with sub-directories) for all customizations.
db/local.php and themes of course but also
https://doc.tiki.org/System+Configuration
Some ideas here: https://dev.tiki.org/File+and+directory+structure+revamp


A feature where minor releases can be automatically/user initiated
upgraded for security and fixes.
---------------------------------------------------------------------------------------------------------------------------------------------
Some notes on this topic here: https://dev.tiki.org/Automatic+Updates

Also, you can do with TRIM: https://doc.tiki.org/TRIM#To_setup_automated_updates


What if we include a folder with all the bundled libraries
------------------------------------------------------------------------------
How would this help with the license issues? If we distribute in a
.zip, we need to respect the license. The idea of letting the site
admin install them is that is that they do an action and thus, they
are agreeing to the license of that lib.

I see how this could improve security if properly done. But it could
add complexity. And security could actually be reduced as these files
have to be writable by Apache to move them from the inactive to active
folder. Which is why I pondering the feature check in the PHAR.


Best regards,

M ;-)

On Sun, Jan 15, 2017 at 7:55 PM, Brendan Ferguson <[hidden email]> wrote:

> Entertain me for a minute. My thoughts about this are evolving.
>
> What if we made ALL libraries treated the same way:
> upgradable/installable/deletable. Libraries being used (based on enabled
> preferences/core functionality) can be blocked from uninstalling. Some of
> these libraries we can bundle and have default options as installed, but in
> essence it they be treated exactly the same.
>
> I would also love to see a feature where minor releases can be
> automatically/user initiated upgraded for security and fixes.
>
> A lot of intranets with no internet access wont be able to download
> packages. So what if we include a folder with all the bundled libraries, so
> they are not installed, but are there..... It wont save bandwidth, but it
> will:
> *Add ease of installing libs with licenses that are incompatible
> *Allow fast security upgrades
> *Prevent security issues in libs that are not being used
> *Make managing tiki via FTP easier (more on that in a bit)
>
> In such a case, tiki could always, by default install the included zip,
> rather than checking for a download. This would mean minimal fussing when
> features are turned on. It would almost always ensure that minimal fussing
> is needed when turning on features. If there is an update for the lib, it
> could be handled ether through an auto/user-initeated process.
>
> How will that work with upgrades.... while Im guessing we may need a
> separate list packaged with each update, that has the minimum requirements
> of each vendor, there will be major release updates here. Part of the
> upgrade-install process could be to check for vendor version requirements
> and install the packaged lib if its needed. Any libraries needed for core
> functionality should of course not be included in zip, in favor of being
> already installed.
>
> Coming back to the FTP. Its impossible to upload a deflated version of tiki
> with ftp. At lest not on any connection ive ever used. The issue is the
> number of files, rather than the size. If we can reduce the vendors to zips,
> so its only tiki files left, it might just be doable, or at least a lot
> closer. Deleting tiki through a FTP program will also WAY faster.
>
> Including the files will also make things like tiki show instances much
> easier. Im guessing there may be a lot of errors if the files needed to be
> downloaded, and the user does not have access to FTP in those circumstances.
>
> Including the zips would also prevent the issue of a lib not being available
> when its needed. What if a server does down (even for 10 minutes) It could
> be a real hassle.
>
> We could also (if its needed) to have a revert button to revert a upgraded
> back to the one included originally with tiki.
>
> I am also guessing that including the libs might make tiki profiles work
> better, where there can be a lot of preferences turned on at once.
>
> Im guessing though that it would be posable to release a version of tiki
> without the bundled packages, but im really not convinced that everything
> will work well without fussing over the vendors control panel, and possibly
> manual downloads and FTP to enable tiki features.... or even failed attempts
> in some cases. Not all admins will be able to FTP/Shell
>
> I do have a question as to how this will effect the "Tiki Security Check"...
> I have not put much thought into it, but its likely major changes will need
> to take place there.
>
> That is as far as my thoughts have gone on this.
>
> On Mon, Jan 16, 2017 at 7:08 AM, Ricardo Melo <[hidden email]> wrote:
>>
>> Hi all,
>>
>> I've been working on a propose regarding the way composer is used today in
>> Tiki and if we can use composer / some kind of packaging to address a couple
>> of use cases that we face today.
>>
>> My proposal is available in:
>> https://dev.tiki.org/Composer+Dependencies+Revamp
>>
>> And the original considerations / goals around this topic are in :
>> https://dev.tiki.org/Install+optional+libraries+in+Tiki+via+Composer
>>
>> From them there are a couple of different use cases I would like to
>> highlight, that I think this Revamp can help address:
>>
>> * If people add (locally) a library to a tiki installation avoid merge
>> conflicts
>> * Allow people to install packages, that because of the licence, can't be
>> bundled with tiki (mPDF is a good example)
>> * Avoid bundle with Tiki HUGE packages that might not be used/activated in
>> most tiki installations (and still support easy installation of the packages
>> to those without shell access to server - only FTP or similar)
>>
>> There is wide range of different approaches based on composer, I've
>> highlighted not only my suggestion but also some other options that are used
>> by other projects but may not be ideal for tiki.
>>
>> I would deeply appreciate that you take a couple of minutes to review the
>> document, feel free to edit if anything don't seem ok and/or raise questions
>> here on the mailing list (I'll do my best to answer them as soon as
>> possible).
>>
>> The goal is to implement the outcome of this discussion for Tiki 17, so I
>> really appreciate your input on this topic.
>>
>> Thank you very much,
>> Ricardo
>>
>>
>> ------------------------------------------------------------------------------
>> Developer Access Program for Intel Xeon Phi Processors
>> Access to Intel Xeon Phi processor-based developer platforms.
>> With one year of Intel Parallel Studio XE.
>> Training and support from Colfax.
>> Order your platform today. http://sdm.link/xeonphi
>> _______________________________________________
>> TikiWiki-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>
>
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> TikiWiki-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>



--
Marc Laporte

http://WikiSuite.org
http://PluginProblems.com
http://Avan.Tech

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Composer Dependencies Revamp

Dr. Sassafras
Im hesitant to move to phar-ing tiki.


I am also not entirely sure that for the reason of installation alone, using some kind of archive format is worth the trouble, but it's certainly a consideration. Every extra file we add to tiki makes installation that much harder in that regard. The reverse is also true.

I think Tiki would need some work to get there. Notably:
A single directory (with sub-directories) for all temp and cache content
A single directory (with sub-directories) for all customizations.
db/local.php and themes of course but also
https://doc.tiki.org/System+Configuration
Some ideas here: https://dev.tiki.org/File+and+directory+structure+revamp

Ya. This should really be done. I've a little effort toward its goal. It makes a lot of things a little easier if /temp and /storage are all in one place. 

That is interesting. I will look more into it. Just looking at the page, it looks like it might only work with multitiki? 

https://www.google.ca/amp/s/amp.reddit.com/r/PHP/comments/13uwgk/phar_performance/?client=safari------
How would this help with the license issues? If we distribute in a
.zip, we need to respect the license.

If vendor libs can be installed from zips when features are turned on it helps in 2 ways.
1, It brings the possibility of tiki automatically downloading the library when the feature is enabled.
2, even without this functionality it makes installing a non-included library almost as easy as dropping the zip into the right folder. 

This could potentially open up a whole new possibility of using non-compatible licensed libs in tiki.

I see how this could improve security if properly done. But it could
add complexity. And security could actually be reduced as these files
have to be writable by Apache to move them from the inactive to active
folder. Which is why I pondering the feature check in the PHAR.

Those files would be fine as read-only. It's the deflated location that would need to be writable. For those who are uncomfortable with Apache having wright access, they can disable it in a production environment. However the vast majority of hacks are enabled when code with known vulns is being used. That makes the biggest practical security issue: how do we make it easier to upgrade? We're doing our part in tiki already. We stay on top of security and release regular security updates.... we send out emails to let people know and we remind them with automatic checks in the admin panel, but there is no guarantee that people will upgrade. The process of an upgrade on a production server is still huge. It often takes me all day (with a customized tiki) to install and test. If I had a button that could install a new vendor update, then revert that update in a moments notice, while it would make a whole day project into a 10 minute job. (Ya, maybe 10 minutes for each vendor update)

Adding code to make this possible is unavoidable. But overall complexity is I think still up for debate. It might make mods and addons obsolete, so it might actually reduce the overall code.... doubtful though.

We're also hesitant in tiki to include large libs. This could be a solution for that. Large libs could be downloaded when needed, and not distributed with tiki. If some obscure feature needs a 20mb lib, no problem! The lib just won't ship with tiki. Right now I think the solution is to include it as a mod. 

So there is a few more thoughts. 

Brendan



Best regards,

M ;-)

On Sun, Jan 15, 2017 at 7:55 PM, Brendan Ferguson <[hidden email]> wrote:
Entertain me for a minute. My thoughts about this are evolving.

What if we made ALL libraries treated the same way:
upgradable/installable/deletable. Libraries being used (based on enabled
preferences/core functionality) can be blocked from uninstalling. Some of
these libraries we can bundle and have default options as installed, but in
essence it they be treated exactly the same.

I would also love to see a feature where minor releases can be
automatically/user initiated upgraded for security and fixes.

A lot of intranets with no internet access wont be able to download
packages. So what if we include a folder with all the bundled libraries, so
they are not installed, but are there..... It wont save bandwidth, but it
will:
*Add ease of installing libs with licenses that are incompatible
*Allow fast security upgrades
*Prevent security issues in libs that are not being used
*Make managing tiki via FTP easier (more on that in a bit)

In such a case, tiki could always, by default install the included zip,
rather than checking for a download. This would mean minimal fussing when
features are turned on. It would almost always ensure that minimal fussing
is needed when turning on features. If there is an update for the lib, it
could be handled ether through an auto/user-initeated process.

How will that work with upgrades.... while Im guessing we may need a
separate list packaged with each update, that has the minimum requirements
of each vendor, there will be major release updates here. Part of the
upgrade-install process could be to check for vendor version requirements
and install the packaged lib if its needed. Any libraries needed for core
functionality should of course not be included in zip, in favor of being
already installed.

Coming back to the FTP. Its impossible to upload a deflated version of tiki
with ftp. At lest not on any connection ive ever used. The issue is the
number of files, rather than the size. If we can reduce the vendors to zips,
so its only tiki files left, it might just be doable, or at least a lot
closer. Deleting tiki through a FTP program will also WAY faster.

Including the files will also make things like tiki show instances much
easier. Im guessing there may be a lot of errors if the files needed to be
downloaded, and the user does not have access to FTP in those circumstances.

Including the zips would also prevent the issue of a lib not being available
when its needed. What if a server does down (even for 10 minutes) It could
be a real hassle.

We could also (if its needed) to have a revert button to revert a upgraded
back to the one included originally with tiki.

I am also guessing that including the libs might make tiki profiles work
better, where there can be a lot of preferences turned on at once.

Im guessing though that it would be posable to release a version of tiki
without the bundled packages, but im really not convinced that everything
will work well without fussing over the vendors control panel, and possibly
manual downloads and FTP to enable tiki features.... or even failed attempts
in some cases. Not all admins will be able to FTP/Shell

I do have a question as to how this will effect the "Tiki Security Check"...
I have not put much thought into it, but its likely major changes will need
to take place there.

That is as far as my thoughts have gone on this.

On Mon, Jan 16, 2017 at 7:08 AM, Ricardo Melo <[hidden email]> wrote:

Hi all,

I've been working on a propose regarding the way composer is used today in
Tiki and if we can use composer / some kind of packaging to address a couple
of use cases that we face today.

My proposal is available in:
https://dev.tiki.org/Composer+Dependencies+Revamp

And the original considerations / goals around this topic are in :
https://dev.tiki.org/Install+optional+libraries+in+Tiki+via+Composer

From them there are a couple of different use cases I would like to
highlight, that I think this Revamp can help address:

* If people add (locally) a library to a tiki installation avoid merge
conflicts
* Allow people to install packages, that because of the licence, can't be
bundled with tiki (mPDF is a good example)
* Avoid bundle with Tiki HUGE packages that might not be used/activated in
most tiki installations (and still support easy installation of the packages
to those without shell access to server - only FTP or similar)

There is wide range of different approaches based on composer, I've
highlighted not only my suggestion but also some other options that are used
by other projects but may not be ideal for tiki.

I would deeply appreciate that you take a couple of minutes to review the
document, feel free to edit if anything don't seem ok and/or raise questions
here on the mailing list (I'll do my best to answer them as soon as
possible).

The goal is to implement the outcome of this discussion for Tiki 17, so I
really appreciate your input on this topic.

Thank you very much,
Ricardo


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel




--
Marc Laporte

http://WikiSuite.org
http://PluginProblems.com
http://Avan.Tech

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Composer Dependencies Revamp

Marc Laporte-3
below


>
> Also, you can do with TRIM:
> https://doc.tiki.org/TRIM#To_setup_automated_updates
>
>
> That is interesting. I will look more into it. Just looking at the page, it
> looks like it might only work with multitiki?
>


No, it's an alternative. I clarified the docs:

https://doc.tiki.org/TRIM#Alternatives

Thanks!

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel