Lessons from Implementing from Scratch

So the last two days I was sprinting at PyCon CA. My original intent was to investigate tools for formalizing experiments with machine learning and reinforcement learning in particular. What we started with (and in the end, ended with) was a kind of "implement reinforcement (Deep Q) learning from scratch" project, where we tried to reconstruct my memories of Deep Q as a simple stand-alone solution and then spent the rest of the time debugging the result.

We had lots of moments where we wound up seeing no learning due to dumb/simple/avoidable errors (mostly mine). At least some of those were due to us trying to build up the solution from scratch level-by-level. That is, we tried to do first a random agent, then an agent that just mapped state directly to best-action... and oops, turns out that one is really just not going to work. The Q function really is needed to make the problem tractable, and we lost a lot of time trying to get to a stable point before going to the final solution. We were seeing lots of "passing" solutions with a straight random search (and a half a dozen random searches that were not intentionally random searches, but were actually vanishing or exploding gradients), but obviously the end of that process was junk. We also had our epsilon gradient calculation wrong, which meant that we'd just stop exploring way too early...

At the end of the sprint, it really does seem I need to focus on more formalizing and providing testing/insight during experiments. So many of the problems should have been obvious if we'd had tooling to show that given fields and values were being set to 0 or 1. At two different points we were debugging for a significant amount of time just to discover we had 0 records being passed into the training due to a missed append call. Dumb/simple metrics, insights into the weight tables (so you can see whether patterns are forming or just dropping to a single value), historic trial data to show you progress/regressions, etc. etc.

Comments

  1. aroma diffuser factory

    aroma diffuser factory on 12/05/2018 7:08 p.m. #

    Fantastic blog! Do you have any tips for aspiring writers?
    I'm hoping to start my own sitge soon but I'm a little lost on everything.

    Would you advise starting with a free platform like Wordpress or goo for a paid option? There
    are so many options out there that I'm completely overwhelmed ..
    Any recommendations? Appreciate it!

  2. Keto Slim Max Reviews

    Keto Slim Max Reviews on 12/05/2018 9:54 p.m. #

    Now I am going to do my breakfast, after having my breakfast coming
    again to read more news. http://spectra.beta.threadgroup.com/ActivityFeed/MyProfile/tabid/57/UserId/142500/Default.aspx

  3. www.iamsport.org

    www.iamsport.org on 12/05/2018 10:36 p.m. #

    Hi to every body, it's myy first go to see of this
    blog

  4. www.iamsport.org

    www.iamsport.org on 12/05/2018 10:36 p.m. #

    Hi to every body, it's my first go to see of this blog

  5. iron thunder hickory nc

    iron thunder hickory nc on 12/05/2018 11:50 p.m. #

    Do you have a spam problem on this blog

  6. bukadepo.com

    bukadepo.com on 12/06/2018 12:31 a.m. #

    Right here is the right webpage for everyone who would like to understand this topic.
    You understand a whole lot its almost tough to argue with you (not
    that I personally would want to…HaHa). You certainly put
    a brand new spin on a subject which has been written about for decades.
    Great stuff, just excellent!

  7. bukadepo.com

    bukadepo.com on 12/06/2018 12:31 a.m. #

    Right here is the right webpage for everyone who would
    like to understand this topic. You understand
    a whole lot its almost tough to argue with you (not that I personally would want to…HaHa).
    You certainly put a brand new spin on a subject which has been written about for decades.
    Great stuff, just excellent!

  8. moneyyellow.net

    moneyyellow.net on 12/06/2018 1:44 a.m. #

    Hi there just wanted to give you a quick heads up. The text in your article seem to be running off
    the screen in Chrome. I'm not sure if this is a format issue or something
    to do with web browser compatibility but I thought I'd post to
    let you know. The style and design look great though!

    Hope you get the issue solved soon. Cheers

  9. moneyyellow.net

    moneyyellow.net on 12/06/2018 1:44 a.m. #

    Hi there just wanted to give you a quick heads up. The text in your article seem
    to be running off the screen in Chrome. I'm not sure if
    this is a format issue or something to do with web browser compatibility but I thought
    I'd post to let you know. The style and design look great though!
    Hope you get the issue solved soon. Cheers

  10. Jack

    Jack on 12/06/2018 11:48 a.m. #

    Wow, this piece of writing is nice, my younger sister is analyzing such things, so I am going to convey her.

    I will immediately seize your rss as I can not in finding your e-mail subscription link or newsletter service.
    Do you’ve any? Please permit me know so that I may subscribe.
    Thanks. Ahaa, its fastidious conversation regarding this piece of writing at this place at this webpage, I have read all that, so now me also commenting here.
    http://cspan.co.uk/

  11. Jack

    Jack on 12/06/2018 11:48 a.m. #

    Wow, this piece of writing is nice, my younger sister is analyzing such things, so I am going to convey her.
    I will immediately seize your rss as I can not in finding your e-mail subscription link or newsletter service.
    Do you’ve any? Please permit me know so that I may subscribe.
    Thanks. Ahaa, its fastidious conversation regarding this piece of writing at this place at this webpage, I
    have read all that, so now me also commenting here.

    http://cspan.co.uk/

  12. Charline

    Charline on 12/06/2018 4:41 p.m. #

    What's up mates, good post and fastidious arguments commented at this place, I
    am actually enjoying by these.

  13. Charline

    Charline on 12/06/2018 4:41 p.m. #

    What's up mates, good post and fastidious arguments commented at this place, I am actually enjoying by these.

  14. the best show youtube attic

    the best show youtube attic on 12/06/2018 9:49 p.m. #

    An interest in playing older video games.

  15. เสื้อโปโล

    เสื้อโปโล on 12/07/2018 2:17 a.m. #

    This is a topic that is close to my heart... Cheers! Where are
    your contact details though?

  16. เสื้อโปโล

    เสื้อโปโล on 12/07/2018 2:17 a.m. #

    This is a topic that is close to my heart... Cheers! Where are your contact details though?

  17. TesadsPa

    TesadsPa on 12/07/2018 5:36 a.m. #

    Игры для девочек, между прочим, это отнюдь не лишь только веселье и способ скоротать скучное время и отвлечься от домашних и утомительных забот. Мини игры <a href=https://www.youtube.com/watch?v=4yiqATrAdhw>красный шарик РЅР° РґРІРѕРёС…</a> это между прочим и просвещающий инструментарий который создает условия чтоб делать превосходнее вдумчивость, взаимодействие и конечно же нарабатывать обязательно важные умения. Помимо прочего Компьютерные игры дозволяют добыть настоящие доллары, рядовой пример - это делают модные игроки.
    <a href=https://www.youtube.com/watch?v=KzcDBntXE6U><img src="https://i.ytimg.com/vi/KzcDBntXE6U/hqdefault.jpg?sqp=-oaymwEZCPYBEIoBSFXyq4qpAwsIARUAAIhCGAFwAQ==&rs=AOn4CLCZJ21Grt6Et_a2l3oT0swuldKF-Q"></a>
    <a href=https://www.youtube.com/embed/cm-glyHw-Tw><img src="https://i.ytimg.com/vi/7rOiVaLfoTg/hqdefault.jpg?sqp=-oaymwEZCPYBEIoBSFXyq4qpAwsIARUAAIhCGAFwAQ==&rs=AOn4CLBy6-f7ONdzQDKJmqhquIxJtEr9zA"></a>
    <a href=https://www.youtube.com/embed/8ZM0s518uvc><img src="https://i.ytimg.com/vi/iL0DqGJ4WkU/hqdefault.jpg?sqp=-oaymwEZCPYBEIoBSFXyq4qpAwsIARUAAIhCGAFwAQ==&rs=AOn4CLCQoQ_ToGZu03t0AXbCX2lsAmJKDA"></a>
    <a href=https://www.youtube.com/embed/_caqD-pFRrI><img src="https://i.ytimg.com/vi/_caqD-pFRrI/hqdefault.jpg?sqp=-oaymwEZCPYBEIoBSFXyq4qpAwsIARUAAIhCGAFwAQ==&rs=AOn4CLDIxn8NUoj4UI6wzT_PVZqsPfW3XQ"></a>
    <a href=https://www.youtube.com/watch?v=uwXANprNcE0><img src="https://i.ytimg.com/vi/4yiqATrAdhw/hqdefault.jpg?sqp=-oaymwEZCPYBEIoBSFXyq4qpAwsIARUAAIhCGAFwAQ==&rs=AOn4CLA1WCwhg-DZm-Cevs49jEWHdD-7zQ"></a>

  18. ketogenic diet pros and cons

    ketogenic diet pros and cons on 12/07/2018 5:59 a.m. #

    Its such as you learn my mind! You seem to understand a lot about this, like
    you wrote the e-book in it or something. I believe that you just could do with a few % to force
    the message home a little bit, however other than that,
    that is excellent blog. A fantastic read. I will definitely be back.

  19. ketogenic diet pros and cons

    ketogenic diet pros and cons on 12/07/2018 5:59 a.m. #

    Its such as you learn my mind! You seem to understand
    a lot about this, like you wrote the e-book in it or something.
    I believe that you just could do with a few %
    to force the message home a little bit, however other than that,
    that is excellent blog. A fantastic read.
    I will definitely be back.

  20. best pills for muscle gain

    best pills for muscle gain on 12/07/2018 8:51 a.m. #

    My brother suggested I might like this blog. He was totally right.

    This post truly made my day. You can not imagine just how much time I
    had spent for this info! Thanks!

  21. best pills for muscle gain

    best pills for muscle gain on 12/07/2018 8:51 a.m. #

    My brother suggested I might like this blog. He was totally right.
    This post truly made my day. You can not imagine just how much time I had spent for this info!
    Thanks!

  22. https://fpvdroneplanenews1987.wordpress.com

    https://fpvdroneplanenews1987.wordpress.com on 12/07/2018 10:04 a.m. #

    I really like your blog.. very nice colors & theme. Did you make this
    website yourself or did you hire someone to do
    it for you? Plz answdr back as I'm looking to construct my oown blog and would like to know
    where u got this from. cheers

  23. https://fpvdroneplanenews1987.wordpress.com

    https://fpvdroneplanenews1987.wordpress.com on 12/07/2018 10:05 a.m. #

    I really like your blog.. very nice colors & theme.
    Did you make thuis website yourself or diid you hire
    someone too do it for you? Plz answer back as I'm looking to construct my own blog and would
    like to know where u got this from. cheers

  24. credit card generator app

    credit card generator app on 12/07/2018 11:22 a.m. #

    We stumbled over here different website and thought I may as well check things out.

    I like what I see so now i'm following you. Look forward to checking out your web page
    for a second time.

  25. credit card generator app

    credit card generator app on 12/07/2018 11:22 a.m. #

    We stumbled over here different website and thought I may
    as well check things out. I like what I see so now i'm following you.
    Look forward to checking out your web page for a second time.

  26. buy sell trade iowa

    buy sell trade iowa on 12/07/2018 11:50 a.m. #

    My brother recommended I may like this blog.
    He used to be entirely right. This post truly made my day.
    You can not believe just how so much time I had spent for this information! Thank
    you!

  27. buy sell trade iowa

    buy sell trade iowa on 12/07/2018 11:50 a.m. #

    My brother recommended I may like this blog. He used to be entirely
    right. This post truly made my day. You can not believe just how so
    much time I had spent for this information! Thank you!

  28. credit card generator hack

    credit card generator hack on 12/07/2018 2:20 p.m. #

    Hello to every one, since I am in fact eager of reading
    this website's post to be updated regularly. It carries fastidious data.

  29. credit card generator hack

    credit card generator hack on 12/07/2018 2:20 p.m. #

    Hello to every one, since I am in fact eager
    of reading this website's post to be updated regularly.

    It carries fastidious data.

  30. credit card info free

    credit card info free on 12/07/2018 3:17 p.m. #

    Greetings! Very useful advice within this post!
    It is the little changes which will make the most significant changes.
    Thanks for sharing!

  31. credit card info free

    credit card info free on 12/07/2018 3:17 p.m. #

    Greetings! Very useful advice within this post!
    It is the little changes which will make the most
    significant changes. Thanks for sharing!

  32. belt conveyor

    belt conveyor on 12/08/2018 1:02 a.m. #

    Wonderful web site. Lots of useful info here. I'm sending it to a few pals ans also sharing
    in delicious. And of course, thank you on your sweat!

  33. belt conveyor

    belt conveyor on 12/08/2018 1:02 a.m. #

    Wonderful web site. Lots of useful info here.
    I'm sending it to a few pals ans also sharing in delicious.
    And of course, thank you on your sweat!

  34. Judi Slot Online

    Judi Slot Online on 12/08/2018 1:39 a.m. #

    hey there and thank you for your info – I have certainly picked up
    something new from right here. I did however expertise
    some technical points using this site, since I
    experienced to reload the site a lot of times previous to I could get it
    to load properly. I had been wondering if your hosting is OK?
    Not that I'm complaining, but slow loading instances times will sometimes affect your
    placement in google and could damage your high-quality score if ads and marketing with Adwords.
    Anyway I'm adding this RSS to my email and can look out for a
    lot more of your respective exciting content. Ensure that
    you update this again soon.

  35. Judi Slot Online

    Judi Slot Online on 12/08/2018 1:39 a.m. #

    hey there and thank you for your info – I have
    certainly picked up something new from right here.
    I did however expertise some technical points using this
    site, since I experienced to reload the site a lot of times previous to I could get it to load properly.
    I had been wondering if your hosting is OK? Not that I'm
    complaining, but slow loading instances times will sometimes affect your placement in google and could
    damage your high-quality score if ads and marketing with Adwords.
    Anyway I'm adding this RSS to my email and can look out for a lot more of your respective exciting content.
    Ensure that you update this again soon.

  36. credit card generator malaysia

    credit card generator malaysia on 12/08/2018 7:35 a.m. #

    Very nice post. I simply stumbled upon your blog and wanted
    to say that I've truly loved browsing your blog posts.
    In any case I will be subscribing on your rss feed and I'm hoping you write again very
    soon!

  37. credit card generator malaysia

    credit card generator malaysia on 12/08/2018 7:35 a.m. #

    Very nice post. I simply stumbled upon your blog and
    wanted to say that I've truly loved browsing your blog posts.
    In any case I will be subscribing on your rss feed and I'm
    hoping you write again very soon!

  38. CliffSag

    CliffSag on 12/08/2018 11:18 p.m. #

    Joe Bonamassa is a famous country singer, so don't miss the possibility to visit [url=http://joebonamassatour.com/]Joe Bonamassa tour [/url]

  39. buy sell trade madison indiana

    buy sell trade madison indiana on 12/09/2018 3:14 a.m. #

    Attractive section of content. I simply stumbled upon your site and in accession capital to say that I get in fact enjoyed account your blog posts.
    Any way I'll be subscribing in your feeds and even I
    achievement you get entry to persistently fast.

  40. buy sell trade madison indiana

    buy sell trade madison indiana on 12/09/2018 3:14 a.m. #

    Attractive section of content. I simply stumbled upon your site and in accession capital to say
    that I get in fact enjoyed account your blog posts.
    Any way I'll be subscribing in your feeds and even I achievement you get entry to
    persistently fast.

  41. credit card generator

    credit card generator on 12/09/2018 9:39 a.m. #

    I enjoy what you guys tend to be up too. This type of clever
    work and exposure! Keep up the good works guys I've you guys to my own blogroll.

  42. credit card generator

    credit card generator on 12/09/2018 9:39 a.m. #

    I enjoy what you guys tend to be up too. This type of clever work and
    exposure! Keep up the good works guys I've
    you guys to my own blogroll.

  43. bin checker website

    bin checker website on 12/10/2018 12:10 a.m. #

    A motivating discussion is definitely worth comment.
    I do think that you need to write more about this subject,
    it may not be a taboo subject but usually people do not speak about
    these subjects. To the next! Best wishes!!

  44. bin checker website

    bin checker website on 12/10/2018 12:10 a.m. #

    A motivating discussion is definitely worth comment. I do think that you
    need to write more about this subject, it may not be a taboo subject but
    usually people do not speak about these subjects. To the next!
    Best wishes!!

  45. bin codes

    bin codes on 12/10/2018 2:13 a.m. #

    Excellent article. I'm facing some of these issues as well..

  46. indoforum88.com

    indoforum88.com on 12/10/2018 5:04 a.m. #

    These are in fact great ideas in regarding blogging. You have touched some nice points here.
    Any way keep up wrinting.

  47. indoforum88.com

    indoforum88.com on 12/10/2018 5:04 a.m. #

    These are in fact great ideas in regarding blogging.

    You have touched some nice points here. Any way keep up wrinting.

  48. squeaky dog toys for aggressive chewers amazon

    squeaky dog toys for aggressive chewers amazon on 12/10/2018 7:46 a.m. #

    That is your cat's method of exhibiting love.

  49. thoibaokhoahoc.com

    thoibaokhoahoc.com on 12/10/2018 8:06 a.m. #

    Great blog! I am loving it!! Will come back again. I am bookmarking
    your feeds also

  50. thoibaokhoahoc.com

    thoibaokhoahoc.com on 12/10/2018 8:06 a.m. #

    Great blog! I am loving it!! Will come back again. I am bookmarking your
    feeds also

  51. credit card generator valid

    credit card generator valid on 12/10/2018 10:14 a.m. #

    Appreciating the persistence you put into your site and detailed information you present.
    It's good to come across a blog every once in a while that isn't the same unwanted rehashed information. Fantastic read!
    I've bookmarked your site and I'm including your RSS feeds to
    my Google account.

  52. credit card generator valid

    credit card generator valid on 12/10/2018 10:14 a.m. #

    Appreciating the persistence you put into your site and detailed information you present.
    It's good to come across a blog every once in a while that isn't the same unwanted rehashed information. Fantastic
    read! I've bookmarked your site and I'm including your RSS feeds
    to my Google account.

  53. sitecongnghe.com

    sitecongnghe.com on 12/10/2018 3:29 p.m. #

    I really like reading and I conceive this website got some truly utilitarian stuff on it!

Pingbacks

Pingbacks are closed.

Post your comment