I can use your cameras.xml as a basis for the required list, I think.
Do you mean to suggest that each CI build or test will have to download the entire tarball? This might be a little restrictive given the probable size of a final tarball. Depending on how you’re implementing CI (Travis?), it might make more sense to cache these on the build system? For instance, I cache npm modules and the build state of pixls.us between CI builds - this might make sense for what you want to do?
Couldn’t agree more.[quote=“LebedevRI, post:5, topic:2882”]
There needs to be a very simple and friendly web-ui for sample submission.
[/quote]
Absolutely agree.
Fair enough!
Good point. I will write an email to him tonight to find an answer to these questions. I certainly don’t want to duplicate effort if it can be avoided.
Agreed, though right now I’d like to try and solve the (harder imo) problem of getting folks engaged enough to help and upload files. We can certainly re-visit this.
Yes, I figure we’ll be looking at some large datasets, but the good side is likely a smaller number of transfer requirements. That is, I’d be surprised to find folks downloading everything multiple times in a day.
My bigger concern here is to remove these types of problems from the smarter folks who are doing the programming and creating software. I think we can handle the infrastructure so y’all (I am in the southern US) don’t have to worry about it. It should _just work_™ work for you.
At the moment I am trying to break this into some manageable tasks once we figure out some architecture answers. I’d like to keep the site and interaction static if possible, and I think we might be able to do this with the infrastructure we already have in place. It will require some work though.
In basic terms, we need.
- An upload mechanism that is simple and low-effort for someone to participate and upload a file.
Ideally, I’m envisioning a user to be able to see a missing make/model with a “+” or “upload” button right there. Let them pick and upload a file directly to our infrastructure.
- A mechanism for extracting information that we may need to sort/filter with.
- A means for displaying the information and allow downloads.
I’ve taken a quick look at the Amazon AWS Lambda as suggested by @jinxos, which looks like it might be a nice fit. I can build a POST form to push the upload into a bucket, we can trigger the Lambda function to finish up processing for us (including pushing necessary files to support 3).
I’ll be back with more when I get some time to write down my thoughts. Getting late here.