Oct 15, 2010

The AppExchange Security Review Process Is An Opaque Mess

As a developer working on the Force.com platform, I've eagerly anticipated submitting my paid application to AppExchange and selling it to customers.  I've recently completed the product, packaged it up as an AppExchange application, and now I'm ready to go right?



Before being listed as a paid app, I need to complete the AppExchange security review process.  This is a relatively new roughly 3 year old process, implemented by Salesforce to prevent malicious applications from entering the AppExchange.  Last fall they started charging $5000 for the review, so it was probably also a profit maker for SFDC, but in spring 2010 they reduced the price to $300 which is obviously much more palatable.

I fill out a questionnaire with obvious risk vector questions (do I load material from servers outside of force.com? do I store user credentials outside of force.com?) and hit submit. Then I get this long-winded, overwritten explanation of how they have put my package through a heuristic threat detector, and a report about everything it found wrong.  And they also say they'll be in touch within 2 days.

So have I failed the review? Can I appeal what the automated heuristics (notorious for false positive results) reported? What happens in 2 days?

How about telling me what I need to do next Salesforce?

(edited 18 Oct 2010 with clarification re: age of process)

21 comments:

  1. pretty informative stuff here, gonna follow you. :)

    if you ever get the time check out my blog! :D

    ReplyDelete
  2. Be sure to tell us how it goes, at least when they get back to you

    ReplyDelete
  3. Hope it went through. Let us know!
    Supporting you, please help me out too! :)

    ReplyDelete
  4. Looking forward to a follow up. Hope it goes well

    ReplyDelete
  5. from $5000 to $300, and the is they are still making money on it.

    ReplyDelete
  6. @ascadian.devil srsly, when they announced $5000 everyone working on apps was like "!!!!" so it wasnt a surprise that they changed the price

    ReplyDelete
  7. the app game seems to be quite hit-or-miss...

    ReplyDelete
  8. A lot of good information!

    And if you ever have the time, would you mind reading my blog?

    ReplyDelete
  9. interesting, I like this... keep up the good work

    ReplyDelete
  10. fullowin 'n' suppin bro :)

    wtfiniggagun

    ReplyDelete
  11. thats good they dropped the price

    ReplyDelete
  12. Hi Simon,

    I manage the AppExchange security program for salesforce.com, and would like to clarify a few things:
    1. This program has been in place for over 3 years.
    2. Based on your description above, it seems like you submitted a request for security review at which time you were asked a few architecture questions. This submission triggers an automated scan to be run against your package. If issues are identified, you are emailed a detailed report explaining the issues at hand along with guidelines to address it. At this point, you should review the report and take corrective action. Failure to do this will just lead to delays in your approval process.

    While all this is taking place, your request for review gets assigned to a support analyst at salesforce.com who will typically get back to you within 48 hours to collect payment and move the process forward. The last step of this process is where a seasoned security professional will manually review your application to confirm that no security issues exist. When this is done, you will receive an official pass/fail email.

    Hope this helps understand where you are in the process. If you have any questions, feel free to email me at vbadhwar (at) salesforce dot com

    ReplyDelete
  13. @varun - Thanks for the clarification. Here's some constructive criticism for improving the process:

    a) Give developers a roadmap for how the security review proceeds. Similar to what you just gave me, but also showing gates. For instance, how many Criticals, Severes and FYIs from automated report can I pass with?

    b) I've now filled out two sets of questions, one basic and one in-depth. But the in-depth questionnaire included most of the basic questions. This felt redundant and as if the process was broken.

    c) Why do I have to call a single individual to make my payment for the review? This seems like a huge bottleneck, why can't we process this online?

    d) I understand it's difficult to give estimates on how quickly security review process can work, especially ahead of Dreamforce with I assume a large workload, but any transparency and commitment that developers could receive regarding timing of completion would go a long way towards reassuring us that the process is working well.

    Thanks again for your attention!

    ReplyDelete
  14. Thanks for the feedback, we will try to further streamline our process.

    ReplyDelete