Network module V1.01 beta

Network is a new native module for ImpressCMS module built using IPF. Its main purpose is to serve as a personal profile/contact database. Contacts can be filtered by tag (requires installation of the Sprockets module), organisation (requires installation of the partners module) or by country. Each contact has a searchable ‘bio’ field where you can enter a personal profile.

To give you an idea of how it might work, I need to build a targeted (by invitation) contact database of scientists working across many different disciplines, institutions and countries. These people need to be able to find other scientists (and organisations) working on similar issues, so that they can collaborate. I will tag each contact according to their research interest (Sprockets), link them to their home institution (which will have a separate profile in the partners module), and enter a brief searchable bio on them (a description of their main interests, projects and publications in a standard format, gathered via survey form and template).

A visitor to the site will be able to generate a list of contacts working in a particular research field (tag), country, or institution or any combination of these via select box filters, and view their personal profile and contact details. On the admin side, contact lists can also be exported to .csv file.

Like all of my modules it is intended to be curated for quality assurance and value, so there is no front end data entry at this stage (partly because of severe language constraints in my target group, and partly because I don’t want the general public spamming in their CVs).

Ideas for future features include:
* Allowing export of search results to .csv from the front end (would be cool from a networking perspective, but I have some reservations about making the database too easy to scrape).
* Support for exporting contacts to vCard.
* Automatically emailing contacts a request to check/update their details periodically (eg. annually).

It is a beta, and does need a bit of polish on the presentation side (also, I need to verify that it can operate independently of Sprockets and Partners) so it is not quite ready for production use yet.

Any ideas for features or bug reports are appreciated (you can just post them below).

>>> Download Network 1.01 beta (bundled with the latest beta versions of Partners and Sprockets, please use these to test it as older versions may not work with Network).

Topic


Re: Proposal to collect installation statistics on Gone Native modules – have your say

As it stands, the module developers don't need to do anything, since their installation is handled by the core. It knows the module and the action being taken - the other modules are passive in this respect.

Steve Twitter: @skenow Facebook: Steve Kenow

Topic


Re: Proposal to collect installation statistics on Gone Native modules – have your say

That was the name I was after. It looks like the backend for the script is not working correctly at the moment, so we don't have any statistics since some time.

leaving out localhost is a good idea, as it will show up as a site that is installed and reinstalled continually

How this can be integrated with other tools like composer needs to be looked at, if we can define the notification there, once, it would be easier to maintain than asking module builders to add yet another part of code that the core should take care of into their modules



Re: Proposal to collect installation statistics on Gone Native modules – have your say

Back to Simon's original post - the only thing triggering icms_module_Handler::installation_notify() is the core installation process and the system module update. It is constructed to support all module changes (install/uninstall, activate/deactivate, update).

Information sent, as David mentions, are a hash of the site URL, the module name, version, and action. The script receiving the update is intended to handle the timestamp (it currently doesn't).

Other improvements can be made - exclude any installations where the root domain is localhost, for example.

It's nice to see some increased activity on the forums, again.

Steve Twitter: @skenow Facebook: Steve Kenow



Re: Proposal to collect installation statistics on Gone Native modules – have your say

I didn't knew that knplabs and Jleagle exists.

But I have tried a bit and seems that works well (I haven't found time to do something with it):

Once you have a working command line version composer and icms you can run this command:

composer require composer/composer dev-master


This will install composer as standart library into vendors folder.

Now it's possible to execute directly some actions!

First answer here I think is a good enought as starting point -> http://stackoverflow.com/questions/17219436/run-composer-with-a-php-script-in-browser



Re: Proposal to collect installation statistics on Gone Native modules – have your say

for a rundown on the installed components that are managed by composer, you need to parse the composer.lock file, which is a JSON file. That gives you information on the current state of the system.

For information on the packages on packagist, there is a RSS feed, with the option of having a RSS feed per user (impresscms for example). That is not bad, but it's quite restrictive. Setting up a satis repository for things like themes and translations would be a good choice imo.

There is also an API implementation by the guys of knplabs, or by Jleagle, but you can bet that the knplabs one is based heavily on symfony, which we don't want too much of for the moment (too many dependencies, not compatible with our way of coding).
That said, composer is built on Symfony2.

dflydev has an embedded composer library that allows us to use the composer functionality from code, without needing command-line access.

Lots of pieces from the puzzle, and we still need to put them together. Sounds like me getting back from an IKEA shopping trip.



Re: Proposal to collect installation statistics on Gone Native modules – have your say

@fiammybe, composer has also a support per version countinting. However it doesn't display this info no where. See -> https://groups.google.com/forum/#!topic/composer-dev/_gn05vlLCok

I think if we figure out how to use this data, would be for us a better choice instead of creating something new.

I know that it's possible to run own provider similar to Packagist. Maybe by doing so, you will get needed data for you.

Quote:

For the module counting, I would love to have a discussion about how we can integrate this more in our release procedure : when a version is tagged in GitHub, an archive is generated automatically, the version is available on the downloads site, and the next time people's sites start checking for updates, they receive an update notification.



We could use something similar like wordpress does. Why we need to make archives? What we really need is a possibility to search for module and install from admin. And for that we can use also composer. And checking for updates too!

I think the hardest thing here is to link somehow composer data with our icms data on module admin page.



Re: Proposal to collect installation statistics on Gone Native modules – have your say

currently, it's quite rudimentary : we send the hashed icms_url and the version of the core in a post url to a certain address on qc.impresscms.org. The code is in htdocs/libraries/icms/module/handler.php, in the function installation_notify()

For the module counting, I would love to have a discussion about how we can integrate this more in our release procedure : when a version is tagged in GitHub, an archive is generated automatically, the version is available on the downloads site, and the next time people's sites start checking for updates, they receive an update notification.



Re: Proposal to collect installation statistics on Gone Native modules – have your say

I can probably help with a backend (if it's done as a module), but would need to know more about what data is being sent and how.

For my own purposes, I was thinking of a module where you can register a new module/version as an object and then the system just counts how many times they've been reported over a given time frame. Maybe some basic stats on what % of known sites each component is present on.

One way it could be done that would actually give benefit back to site owners is to have stats reporting incorporated into a 'check for updates'. We get a readout of what components are in use, admins get a warning in the control panel if something is out of date and needs patching, perhaps even an email advisory (delivered by their own system).



Re: Proposal to collect installation statistics on Gone Native modules – have your say

Since a few versions (1.3.2 I think) every installation of ImpressCMS has been sending anonymised data of the installed version of ImpressCMS to our servers. Until now, we have not done anything with it, and frankly we lost a packet of that data due to a server migration at the beginning of the year.

We could enhance that data collection, but for that we'd need some support in building a backend that can make sense of it all.

I as well am interested in knowing more about our userbase, and all the widely used CMSes have something similar.

One basic condition would be that it can be turned off easily.

@mekdrop : composer uses packagist in most cases (I'm considering setting up a separate repository for languages and themes, they don't belong on packagist in my opinion), and all you have is number of installations. What would be interesting to have is the bundled info : how many times is the last version of a module installed on which versionof ImpressCMS, and in combination with which other modules? Which PHP versions, etc...



Re: Proposal to collect installation statistics on Gone Native modules – have your say

  • 2015/10/21 10:48:54

i find this idea very good and would even go a step further. make it for ImpressCMS in general. as a part of the installer you could ask if usage statistics reporting should be enabled like the debian project asks for example. i have no problem with such things in general and it leads to a win win situation. an option to enable or disable it at any time would be good. my backup machine doesnt send any statistics for example while my desktop does it a lot. its a help and also has some motivating potential for the dev and the user gets a better overall experience.

it looks like MekDrop suggested that exactly this is happening at the moment, very nice.

Besides this general approach, i for myself would support your private collection of data as well if you whish.



Re: Proposal to collect installation statistics on Gone Native modules – have your say

fiammybe is working on modules instalation with composer for 2.0. Composer already has build-in support for tracking such data. So maybe is a better idea to wait a bit? :)

There is already one module that uses such functionality ->
https://packagist.org/packages/impresscms/impresscms-module-billboard



Proposal to collect installation statistics on Gone Native modules – have your say

Small problem – I have no idea how many people use my “Gone Native” modules, or which ones. None whatsoever. Basically I’m completely in the dark when it comes to allocation of my extremely limited dev time. I don’t want to work on things that nobody finds useful. I’d rather do something else that they might.

So I was thinking of collecting some anonymous installation statistics – but I want to get a sense of whether people are ok with this or not. So here’s how it could work:

1. I would add a small statistic collection routine in the onInstall() and onUpdate() functions of each module. It will run when you initially install a module, and also when you update it (eg. when you upgrade to a new version). It will *not* run at any other time, so there will be zero performance issues.

2. The routine would send the following data to a private collection page on one of my websites:

a. A hash of your domain name (your domain name would *not* collected).

b. The module name.

c. Possibly (maybe not needed), the module version number.

d. Possibly (maybe not needed) the date the record was collected.

A sample record would look like this:

Hash: b879bea0efd070f31249e99df80208ce7e79dd2b31f19ad4904b9d89597c07ad
Module: News
Version: 1.14
Timestamp: 1445411699

Using a hash of the domain name lets you keep your privacy while giving me a way to detect duplicate records (for example, if you update the module a couple of times). It will also let me see which modules are commonly installed together on a site (eg. is the Sprockets tagging module in use or not?).

Now, I have no intentions to force this on anyone, so if people generally don’t like this idea then I won’t pursue it. If most people appear to be ok with it then I will go ahead, but I will also provide instructions on how to disable the statistics collection routine (basically, it would involve commenting out a couple of clearly marked lines of code in onupdate.inc.php).

Just to be clear, at this stage this proposal only concerns my own modules (the “Gone Native” ones). It does not apply to the official project modules or anyone else’s (though perhaps other authors would be interested in collecting their own stats).

If you have any feedback on this proposal, positive or negative, please post your opinion below.



Re: Security of file uploads

Thanks. I ended up adding an upload directory inside the trust path, as it already has a random prefix, and moving the files over using afterSave() is working.

I'll change the lookup parameter to a hash as you suggested and keep the admin check, thanks for that.



Re: Security of file uploads

From what I can remember, when uploading file trought with default ICMS form getUploadDir is used. So, you can reimplement on your object. Also, you can use setImageDir elsewhere (I don't know why but setImageDir sets global upload directory for object).

I think it's easiest why to set that folder.

Overall I think you ware on the right track. But I have few ideas to improve security a bit more :)

You can create use path outside of public ICMS dir.

Using document/author's IDs in URL is bad idea if you want more security - because it's not very hard to find others just by changing parameters. Use hash (sha1 or md5) instead of params combination.



Security of file uploads

I have a project to allow people (registered, but essentially anonymous users) to upload documents for a conference. I also need to let editors (who will be registered and trusted users) access each file as it is submitted via a URL, which they will receive in an automatically generated email alert. I’m looking for advice on how to handle this securely.

At the moment, the default behaviour for file uploads is to:

i) Check the file extension against a list of permitted mimetypes.

ii) Check file size (and for images, width and height) against specified limits.

iii) Rename the file, adding a prefix (timestamp?) to it, which makes it harder to guess.

iv) The file is saved in a /uploads/module_name/object_name directory, which is easy to guess.

v) An index.html file is created in this directory to prevent casual listing of its contents.

vi) The uploaded files acquire CHMOD permissions of 644, at least on my server.

I am wondering if it is possible to change the default upload directory to something less predictable, so that people can’t guess URLs? In the base handler class there is a deprecated method setUploaderConfig() that let you specify the upload path, but the current method enableUpload() doesn’t, as far as I can see.

At the moment I’m thinking about doing it this way:

* Use afterSave() to move uploaded files from the default upload directory to an “unknown” one with a randomised name, or inside the trust path folder, to make it hard or impossible to access files by guessing URLs, and tighten the file permissions.

* Editors (logged-in administrators of the module) receive an email notice when someone submits a document. The email contains a URL to a file lookup page and includes the IDs of the document and its author.

* The file lookup page is only accessible to the editors. The page accepts two arguments, which are the document/author IDs in the emailed URL.

* If the lookup page finds a file matching the author/document ID parameters, it outputs the file via header() so the editor can download it, with a different (standardised) file name.

The idea of the lookup page is basically to prevent all direct access to the files and force people to pass the ‘logged in admin’ test to access them indirectly.

Does this sound reasonable? Is there a better way?



Re: File uploads - which control?

  • 2015/10/16 8:52:49

i like these reads a lot, it gives some fundamental insight and i hope i will grasp how icms is fabricated to do things too. i started to learn to programm with arduino and also got me a good book on php.
at the moment im more the allrounder/technician/designer without a real programming background but this will come now and with time.
at the moment im working on some infrstructure and a good development environment, read about good git branching techniques and allready setup my disaster and recovery plan. i try to get to know how things work and my plan was to directly start working/enjoying/learning/playing around with ImpressCMS 2. In not too long time you will see the first approach to planetimpress.org, a resource for tips tricks downloads but also some kind of diary regarding the usage and development of impresscms 2.

drop me a line any time on g+, twitter or by mail if like, im almost always happy to talk.

have a good day!



Re: File uploads - which control?

Sorry, I was a bit not specified when I was answering. setControl to type file does everything and should be used every time all other logic if needed (like Steve said that such exists) should be handled on core level not on module developer level :)



Re: File uploads - which control?

Thanks Steve.



Re: File uploads - which control?

I haven't used them, not knowingly, but from looking at the different classes, here's what I would assume -

These 2 extend the same class and have a slight, but significant difference:

class icms_ipf_form_elements_Upload extends icms_form_elements_File class icms_ipf_form_elements_File extends icms_form_elements_File

icms_ipf_form_elements_File will also display the current file associated with the object if there is one, along with a form element to upload a new file.

These 2 extend the same class (one of the above), the only difference being in the name of the element when it is rendered:
class icms_ipf_form_elements_Fileupload extends icms_ipf_form_elements_Upload class icms_ipf_form_elements_Imageupload extends icms_ipf_form_elements_Upload


One of them could probably be removed.

These 2 extend the same class (icms_form_elements_Tray). Both also use icms_ipf_form_elements_Fileupload:
class icms_ipf_form_elements_Richfile extends icms_form_elements_Tray class icms_ipf_form_elements_Image extends icms_form_elements_Tray


the Image element also adds a checkbox to delete an existing image, if one exists, and it adds rel=lightbox to the rendering of the image.

There is some duplication that could be simplified down to fewer classes that are more flexible, which is usually the case when multiple classes or functions are added in - it didn't do what someone wanted or needed, so they added something extra.

Steve Twitter: @skenow Facebook: Steve Kenow




 Top