Oct 15, 2010

The AppExchange Security Review Process Is An Opaque Mess

As a developer working on the Force.com platform, I've eagerly anticipated submitting my paid application to AppExchange and selling it to customers.  I've recently completed the product, packaged it up as an AppExchange application, and now I'm ready to go right?



Before being listed as a paid app, I need to complete the AppExchange security review process.  This is a relatively new roughly 3 year old process, implemented by Salesforce to prevent malicious applications from entering the AppExchange.  Last fall they started charging $5000 for the review, so it was probably also a profit maker for SFDC, but in spring 2010 they reduced the price to $300 which is obviously much more palatable.

I fill out a questionnaire with obvious risk vector questions (do I load material from servers outside of force.com? do I store user credentials outside of force.com?) and hit submit. Then I get this long-winded, overwritten explanation of how they have put my package through a heuristic threat detector, and a report about everything it found wrong.  And they also say they'll be in touch within 2 days.

So have I failed the review? Can I appeal what the automated heuristics (notorious for false positive results) reported? What happens in 2 days?

How about telling me what I need to do next Salesforce?

(edited 18 Oct 2010 with clarification re: age of process)
Posted by at

21 comments:

  1. Nerd LifeOctober 15, 2010 at 11:33 PM

    thanks for sharing this!

    ReplyDelete
  2. AnonymousOctober 16, 2010 at 2:18 AM

    pretty informative stuff here, gonna follow you. :)

    if you ever get the time check out my blog! :D

    ReplyDelete
  3. BrandonOctober 16, 2010 at 11:49 AM

    Be sure to tell us how it goes, at least when they get back to you

    ReplyDelete
  4. wishbearOctober 16, 2010 at 2:09 PM

    Hope it went through. Let us know!
    Supporting you, please help me out too! :)

    ReplyDelete
  5. gombyOctober 16, 2010 at 2:10 PM

    Looking forward to a follow up. Hope it goes well

    ReplyDelete
  6. Ascadian.DevilOctober 16, 2010 at 2:21 PM

    from $5000 to $300, and the is they are still making money on it.

    ReplyDelete
  7. MarioOctober 16, 2010 at 2:22 PM

    pretty interesting

    ReplyDelete
  8. SimonOctober 16, 2010 at 2:26 PM

    @ascadian.devil srsly, when they announced $5000 everyone working on apps was like "!!!!" so it wasnt a surprise that they changed the price

    ReplyDelete
  9. CrunkyOctober 16, 2010 at 2:43 PM

    the app game seems to be quite hit-or-miss...

    ReplyDelete
  10. AnonymousOctober 16, 2010 at 3:30 PM

    awesome post!

    ReplyDelete
  11. AdrianOctober 16, 2010 at 4:53 PM

    A lot of good information!

    And if you ever have the time, would you mind reading my blog?

    ReplyDelete
  12. Discerning GentlemanOctober 16, 2010 at 4:56 PM

    interesting, I like this... keep up the good work

    ReplyDelete
  13. razortekOctober 17, 2010 at 3:23 AM

    fullowin 'n' suppin bro :)

    wtfiniggagun

    ReplyDelete
  14. ANIKI: Epitome of ManlinessOctober 17, 2010 at 9:06 AM

    hope you get through your dilemma, man.

    ReplyDelete
  15. DelfinOctober 17, 2010 at 6:50 PM

    thats good they dropped the price

    ReplyDelete
  16. R_DoggOctober 17, 2010 at 6:51 PM

    good luck to you

    ReplyDelete
  17. befashionistic.blogspot.com/October 18, 2010 at 5:03 AM

    yep, mess indeed

    ReplyDelete
  18. AnonymousOctober 18, 2010 at 9:14 AM

    Hi Simon,

    I manage the AppExchange security program for salesforce.com, and would like to clarify a few things:
    1. This program has been in place for over 3 years.
    2. Based on your description above, it seems like you submitted a request for security review at which time you were asked a few architecture questions. This submission triggers an automated scan to be run against your package. If issues are identified, you are emailed a detailed report explaining the issues at hand along with guidelines to address it. At this point, you should review the report and take corrective action. Failure to do this will just lead to delays in your approval process.

    While all this is taking place, your request for review gets assigned to a support analyst at salesforce.com who will typically get back to you within 48 hours to collect payment and move the process forward. The last step of this process is where a seasoned security professional will manually review your application to confirm that no security issues exist. When this is done, you will receive an official pass/fail email.

    Hope this helps understand where you are in the process. If you have any questions, feel free to email me at vbadhwar (at) salesforce dot com

    ReplyDelete
  19. SimonOctober 18, 2010 at 9:24 AM

    @varun - Thanks for the clarification. Here's some constructive criticism for improving the process:

    a) Give developers a roadmap for how the security review proceeds. Similar to what you just gave me, but also showing gates. For instance, how many Criticals, Severes and FYIs from automated report can I pass with?

    b) I've now filled out two sets of questions, one basic and one in-depth. But the in-depth questionnaire included most of the basic questions. This felt redundant and as if the process was broken.

    c) Why do I have to call a single individual to make my payment for the review? This seems like a huge bottleneck, why can't we process this online?

    d) I understand it's difficult to give estimates on how quickly security review process can work, especially ahead of Dreamforce with I assume a large workload, but any transparency and commitment that developers could receive regarding timing of completion would go a long way towards reassuring us that the process is working well.

    Thanks again for your attention!

    ReplyDelete
  20. AnonymousOctober 19, 2010 at 4:31 PM

    Thanks for the feedback, we will try to further streamline our process.

    ReplyDelete
  21. ! WALLPAPERS BY ERDNUSS2 !October 20, 2010 at 12:28 AM

    i really like your blog

    ReplyDelete

Subscribe to: Post Comments (Atom)