Hello Sanket, It's ok to stick on snake example if you are interested most! Ray <https://docs.ray.io/en/master/index.html> is a framework offers scalability and hyperparameters tuning options. In this end, we can add wrappers in deepbots in order to support ray. More specifically, RLlib <https://docs.ray.io/en/master/rllib.html>provides an API in order train models efficiently and in a distributed manner. The major concern about integrating ray in deepbots is the parallelization. For example OpenAI gym uses Vectoralized Enviroments <https://stable-baselines.readthedocs.io/en/master/guide/vec_envs.html>. /Vectorized Environments are a method for stacking multiple independent environments into a single environment. Instead of training an RL agent on 1 environment per step, it allows us to train it on //|n|//environments per step. /When it comes to Webots parallelization can be happened in two different ways. The first one is to create multiple instances of Webots in order to run different environments/. /The second one is to create a grid in Webots world with different runs of the same example (for example a 3x3 grid in which we try to solve 9 cartpole instances). We have made some progress in former case, in which we are using external controller as described in webots documentation.//// <https://cyberbotics.com/doc/guide/running-extern-robot-controllers> // Both ways are equally good, in order to integrate RLLib. Of course this can be an optional task, after complete the examples that you are mentioned in proposal. RLlib can help us to achieve faster convergences and better models in the existing examples Thank you, Manos. P.S As the GSoC platform gives the option to post a draft proposal, you can use it and I will provide you feedback if need be. On 1/4/21 1:07 π.μ., Sanket Thakur wrote: > Hello Manos, > Thank you for your feedback. I have tried to follow up with your > reviews and added few concerns in the proposal itself for you to > comment on. > > Best, > Sanket > > > On Mon, Mar 29, 2021 at 8:00 AM Manos Kirtas <manolis [ dot ] kirt [ at ] gmail [ dot ] com > <mailto:manolis [ dot ] kirt [ at ] gmail [ dot ] com>> wrote: > > Thank you Sanket, > > I have added some comments directly on the PDF. Feel free contact > me in order to discuss them. Excuse me for the delay! > > Best, > > Manos. > > On 23/3/21 7:52 μ.μ., Sanket Thakur wrote: >> Hello Manos, >> Thanks for your reply. >> I have modified my proposal accordingly. We can still iterate >> over it. >> Let me know your thoughts on it. >> >> Best, >> Sanket >> >> On Tue, Mar 23, 2021 at 12:15 PM Manos Kirtas >> <manolis [ dot ] kirt [ at ] gmail [ dot ] com <mailto:manolis [ dot ] kirt [ at ] gmail [ dot ] com>> wrote: >> >> Hello, >> >> Glad to hear that you are interested on contributing in >> deepbots project. I have some comments on you proposal >> >> * It will be helpful if you specify in that you will work >> on first. For example, there is an extensive list of >> examples >> <https://github.com/aidudezzz/deepbots/issues/85> that >> you can work on. Being more specific can help us to guide >> you in order to compose a strong proposal >> * What types of problems you are interested for? What kind >> of robots can be used to replicate those gym examples? >> * Are you interested on contributing RL algorithms? Is >> there any existing implementation that can be used or we >> should develop it from scratch. I totally recommend to >> take on a look on existing implementation (such as >> stable-baselines >> <https://stable-baselines.readthedocs.io/en/master/>) >> * If you are interested on implementing RL algorithms from >> scratch, it would be great to cite the respective papers. >> * Which framework you are going to use in order to >> implement those algorithms (etc. pytorch, tensorflow)? >> * Elaborate as more as possible the custom testbets that >> you are interested to develop. What's you ideas? What >> type of task we want to solve? What robot can be used? >> Can this problem be solved with both discrete and >> continuous action space? >> * I found very interesting idea to have an infrastructure >> for hyperparameter optimization! Can we use an existing >> framework for that, for example ray? <https://ray.io/> >> >> In my prospective it will be better to stick on 3 specific >> examples/tasks in order to further examine what can be used. >> Additionally, I feel that firstly we should take a look on >> existing implementations of RL algorithms and integrate them >> on those specific examples. Stable-baselines are already >> supported from deepbots and can be easily integrated on any >> example with not so much effort. Regarding the hyperparameter >> optimization, I will recommend ray since a have some >> experience and I can guide you. Of course any other ideas it >> is more than welcome to be discussed! >> >> Finally, I find very useful to include a timeline in which >> you can schedule your different ideas and develop a plan that >> can be feasible on the given timeline. >> >> Those some comments can extend our discussion in order to >> develop a strong proposal. I'm glad to hear your thoughts >> about the above comments! >> >> Best regards, >> >> Manos. >> >> >> On 22/3/21 12:35 μ.μ., Sanket Thakur wrote: >>> Hello, >>> I am writing this to express my interest to work on '*Extend >>> deepbots to support stable-baselines and implement gym-style >>> default RL environments *' as a part of Gsoc 2021. >>> I am attaching my proposal for the project and relevant >>> contributions. >>> It'd be great to hear your reviews on it. >>> >>> Thanks. >>> >>> Best, >>> Sanket >>> >>> ---- >>> Λαμβάνετε αυτό το μήνυμα απο την λίστα: Λίστα αλληλογραφίας και συζητήσεων που απευθύνεται σε φοιτητές developers \& mentors έργων του Google Summer of Code - A discussion list for student developers and mentors of Google Summer of Code projects., >>> https://lists.ellak.gr/gsoc-developers/listinfo.html >>> Μπορείτε να απεγγραφείτε από τη λίστα στέλνοντας κενό μήνυμα ηλ. ταχυδρομείου στη διεύθυνση <gsoc-developers+unsubscribe [ at ] ellak [ dot ] gr> <mailto:gsoc-developers+unsubscribe [ at ] ellak [ dot ] gr>. >>
---- Λαμβάνετε αυτό το μήνυμα απο την λίστα: Λίστα αλληλογραφίας και συζητήσεων που απευθύνεται σε φοιτητές developers \& mentors έργων του Google Summer of Code - A discussion list for student developers and mentors of Google Summer of Code projects., https://lists.ellak.gr/gsoc-developers/listinfo.html Μπορείτε να απεγγραφείτε από τη λίστα στέλνοντας κενό μήνυμα ηλ. ταχυδρομείου στη διεύθυνση <gsoc-developers+unsubscribe [ at ] ellak [ dot ] gr>.