13:00:18 <aglitke> #startmeeting
13:00:18 <kimchi-bot> Meeting started Wed Sep  4 13:00:18 2013 UTC.  The chair is aglitke. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:18 <kimchi-bot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:00:24 <aglitke> #meetingname scrum
13:00:24 <kimchi-bot> The meeting name has been set to 'scrum'
13:00:38 <aglitke> #info Agenda
13:01:03 <aglitke> #info Feature rundown for Sept release
13:01:20 <aglitke> Anything else?
13:01:30 <aglitke> hi royce xinding zhoumeina
13:01:33 <aglitke> hi hlwanghl
13:01:38 <ming> hi aglitke
13:01:42 <royce> hi adam!
13:01:43 <hlwanghl> good morning
13:01:47 <aglitke> hi ming :)
13:01:51 <zhoumeina> :)
13:01:53 <xinding> hi
13:01:57 <aglitke> Any other topic besides feature rundown?
13:02:23 <royce> looks good for me, seem we have new fellow, apporc
13:02:27 <ming> rundown means freezing.
13:02:28 <aglitke> I really want to focus on nailing down what is required to get the outstanding features merged soon
13:02:34 <ming> ?
13:02:42 <apporc> yeah, hi
13:02:49 <aglitke> means getting the last important features in.
13:03:05 <ming> approc, could you introduce yourself first?
13:03:07 <aglitke> apporc: Welcome
13:03:48 <zhoumeina> apporc: welcom
13:03:52 <shaohef> hi all
13:04:01 <alinefm> hi people! =)
13:04:03 <ming> aglitke, we have ISO scanning and authentication on going now. I think these two are in the top priority list
13:04:12 <alinefm> missed you last week
13:04:19 <aglitke> yeah let's open the topic up...
13:04:25 <aglitke> #topic eature rundown for Sept release
13:04:32 <aglitke> #topic Feature rundown for Sept release
13:04:39 <hlwanghl> vm edit ready or not？
13:04:44 <aglitke> These are the ones I am aware of:
13:04:45 <AdamKingIT> https://github.com/kimchi-project/kimchi/wiki/Todo-September-Base
13:05:02 <aglitke> These are the most complex ones that need the most attention:
13:05:07 <aglitke> 1) ISO Scanning
13:05:11 <aglitke> 2) ISO Streaming
13:05:18 <aglitke> 3) Authentication
13:05:26 <aglitke> I would like to start with these.
13:05:55 <aglitke> I think we should try to break these up to get small pieces merged.
13:06:02 <aglitke> 1) ISO Scanning
13:06:15 <aglitke> royce: Can you summarize where we stand?
13:06:38 <royce> depend on multi-fix for shallow, depends on refresh and iso generator for deep scan
13:06:51 <royce> shallow's ready I would say
13:06:52 <apporc> just another programmer, love open source  ,linux. btw, iam using an android.  it is too slow to type . please take easy to discuss,and i will just listen.
13:07:13 <royce> deep scan's rfc I'll say
13:08:10 <aglitke> royce: Can you give me the patch subjects for the dependencies?
13:08:20 <aglitke> I am going to add them to the wiki
13:08:21 <ming> royce about the storage pool refersh, we need to split Bingbu's patch
13:08:39 <royce> Of course aglitke, one sec
13:09:19 <aglitke> Are the refresh and iso generator patches ready?  Have they been reviewed?
13:09:34 <aglitke> Is there a UI piece?
13:09:34 <royce> shallow:[project-kimchi][PATCHv3 0/4] Misc bug fix for scan preparation
13:09:42 <ming> royce, based on the discussion we had, there will be no stand-alone refresh API, just list
13:10:01 <royce> right, ming
13:10:06 <shaohef> ming: agree
13:10:13 <hlwanghl> Sorry, connection lost just now
13:10:32 <royce> xingding, could you pleas direct us to the UI patch u sent?
13:10:35 <aglitke> ming: So bingbu's refresh series will need to be reposted?
13:10:53 <ming> aglitek: the patch include other features.
13:10:58 <xinding> I sent out a patch about shallow scan
13:11:05 <ming> agiltke: i will split out it.
13:11:12 <aglitke> xinding: Can you find the patch subject?
13:11:17 <royce> aglitke, iso generator's ready I think
13:11:17 <aglitke> ming: ok, thanks.
13:11:22 <xinding> [PATCH] Create templates by shallow scan
13:11:27 <aglitke> royce: So far I have:
13:11:32 <aglitke> [PATCHv3 0/4] Misc bug fix for scan preparation
13:11:41 <aglitke> Which other series are needed?
13:11:47 <royce> deep: Add pseudo iso generator
13:11:55 <aglitke> I will focus my energy on those
13:12:00 <royce> deep : Update storagepool:
13:12:00 <royce> add refresh action,get vol numbers and available space of a pool
13:12:23 <aglitke> deep is RFC right?  So iso gen is not needed for shallow
13:12:29 <royce> but the refresh one is not ready according to our discussion, need rebase
13:12:35 <royce> right, aglitke
13:12:55 <aglitke> What is the subject for the Shallow scan series?
13:13:06 <aglitke> Then all I need is the UI patch
13:13:21 <royce> [project-kimchi][PATCHv10 0/7] shallow scan support
13:13:26 <ming> royce, no refresh API, just make sure list will refresh the list automatically.
13:13:44 <royce> true, ming
13:14:13 <royce> let me check xingding's UI patch name
13:14:18 <ming> royce, are you requesting to merge deep ISO scan also?
13:14:27 <xinding> There is only one UI patch, [PATCH] Create templates by shallow scan
13:14:46 <aglitke> ok, great.
13:14:46 <xinding> It depends several backend patches
13:14:54 <royce> aglitke:[project-kimchi] [PATCH] Create templates by shallow scan
13:15:30 <aglitke> I am going to put deep scan on a somewhat lower priority while we get shallow working.
13:16:03 <ming> aglitke, deep scanning patchs are there.
13:16:07 <aglitke> I would ask the team to please focus review and testing efforts on the specific patch series we identify in this discussion.
13:16:25 <aglitke> ming: royce has said they are still in RFC condition.
13:16:47 <aglitke> I want to reduce the backlog of outstanding patches and that means taking what is ready first.
13:16:48 <ming> aglitke: NP
13:16:56 <aglitke> 2) ISO Streaming
13:17:03 <aglitke> From my understanding we need:
13:17:13 <royce> aglitke, it is tested, but I'm not sure you guys like the design, but since your effort is limited I can wait until proper time
13:17:19 <aglitke> * Capabilities detection
13:17:32 <royce> s/effort/time
13:17:47 <aglitke> * File-based template storage
13:18:26 <aglitke> * API to list known iso templates
13:18:56 <aglitke> * UI to create a template from a known network iso
13:19:18 <aglitke> * UI to create a template from a user provided URL
13:19:44 <alinefm> this last one already exist
13:19:45 <ming> aglitke:File-based template storage?
13:19:50 <aglitke> * Backend code to generate the proper domain xml for streamed isos
13:20:01 <royce> aglitke, For file based template storage, is that one change from object store to file?
13:20:14 <aglitke> yes, but that needs more discussion here.
13:20:22 <royce> right, aglitke
13:20:57 <aglitke> I think there is a difference between templates and known iso links
13:21:08 <aglitke> even though both have the same json format
13:21:35 <aglitke> Templates are user created and can be deleted
13:22:04 <aglitke> they can also be installed by an external package into /etc/kimchi/templates/
13:22:44 <ming> what the user can do with known iso links?
13:22:54 <aglitke> Templates are accessed by /templates/* URI
13:22:59 <ming> creating VM by iso links directly?
13:23:04 <aglitke> for iso links:
13:23:24 <aglitke> They are supplied by the kimchi package or an add-on package only.
13:23:32 <royce> so with installable templates we still expose iso links?
13:23:37 <aglitke> Then cannot be added by the API or deleted.
13:24:04 <aglitke> They might be located at: /etc/kimchi/distros
13:24:22 <aglitke> they can be accessed by the URI: /config/distros
13:24:33 <aglitke> they can only be listed
13:24:52 <royce> and update? aglitke
13:24:55 <aglitke> to create a template from a known distro the ui would fetch the appropriate distro
13:25:21 <aglitke> and use the json to construct a template json object to post to /templates
13:25:28 <aglitke> no updates to distros via the api
13:25:46 <aglitke> it is static application configuration
13:25:52 <ming> isos are pre-installed before Kimchi by admin?
13:26:00 <aglitke> can be edited by an external package or by the administrator.
13:26:09 <aglitke> ming: Just the iso information
13:26:22 <royce> what if we want to use a local mirror iso link? I assume we need to change the default official one?
13:26:25 <aglitke> like links to the remote file, icon, and recommended vm settings.
13:26:36 <ming> aglitke, so we need tools to create iso information.
13:26:44 <aglitke> we would drop a new file into /etc/kimchi/distros
13:27:07 <aglitke> ming: These would not be commonly edited by end users.
13:27:16 <alinefm> the iso link wil be what? a xm file with the information or only the iso file itself/
13:27:17 <alinefm> ?
13:27:39 <aglitke> in /etc/kimchi/distros would be a series of .json files
13:27:49 <ming> aglitke, I think even administrator need a handy tool to change those distro informations.
13:28:00 <aglitke> the format is a list of json objects:
13:28:02 <aglitke> [
13:28:53 <aglitke> { 'name': 'Fedora 19', 'os_distro': 'fedora', 'os_version': '19', 'cdrom': 'http://...', ... },
13:28:54 <aglitke> ]
13:29:13 <aglitke> Notice the format of the json object matches the format of a VM Template.
13:29:42 <aglitke> to make a template from a distro, the UI just echos the object into a new json object and substitutes 'name' with the desired template name.
13:29:46 <alinefm> right! so the ui will be able to list those iso links and then the user can select one to create a template from?
13:29:49 <royce> OK, read only iso info does not need to care about locks, but template file need, I think we need to tackle the issue of transaction write and r/w serialization of file
13:29:58 <aglitke> alinefm: yes
13:29:59 <alinefm> or the user will be able to create a vm from the iso link directly?
13:30:34 <aglitke> alinefm: If the user specifies a link by themselves, they also need to select a distro and version (or unknown)
13:31:12 <AdamKingIT> But that would be to create a template from a link directly. I don;t think we are contemplating creating VMs through any path other than template
13:31:15 <royce> xingding, AdamKingIT so the UI will still needed for this one for creating template
13:31:30 <aglitke> royce: I responded to your question on the list... The ObjectStore already uses a session / transaction lock that is global to the objectstore.
13:31:48 <shaohef> aglitke:  the cdrom can be several links? for http://iso.us.com http://iso.cn.com   ftp:// ?
13:31:49 <ming> royce, are you talking abot the administrator can change the template with writing operations?
13:31:58 <aglitke> AdamKingIT: Right, create template from link. then create vm from template.
13:32:15 <aglitke> shaohef: For now it is one link.
13:32:22 <royce> ming, yes
13:32:59 <aglitke> royce: class ObjectStoreSession(object):
13:33:26 <royce> aglitke, let me take a look...
13:33:30 <aglitke> This gives you safe, exclusive access to the ObjectStore.
13:33:42 <xinding> royce: Got it.
13:33:51 <AdamKingIT> The UI for this as a baseline will need to cover creating a template from the avail 'known distros', where the use picks from the distros
13:34:26 <AdamKingIT> After we have that, we'll eventually want to add 'create a temple from a URL' where the user has to input the URL
13:34:38 <aglitke> yep
13:35:06 <xinding> create a temple from a URL, we already have this
13:35:33 <aglitke> maybe that UI could redirect right to the template edit window
13:35:42 <aglitke> after it is created.
13:36:06 <alinefm> AdamKingIT, the UI create template passing an url already exists
13:36:15 <xinding> But maybe not only one templates could be created one time
13:36:24 <royce> aglitke, so one time for a r/w access from the user of objstore?acceptable, for transaction write we still want to make sure it never writes part of the file
13:37:16 <xinding> If user select several isos, it's difficult to decide to redirect to which edit window
13:37:41 <aglitke> xinding: I only mean to do the redirect when adding an iso from a single custom URL
13:37:59 <xinding> aglitke: got it
13:38:04 <aglitke> for the UI where the user selects known distros there is no need to redirect.
13:38:05 <shaohef> royce: aglitke: a tool to write the iso json file?  such as the ajax shell ?
13:38:23 <aglitke> Maybe eventually but it's not important for September.
13:39:08 <shaohef> aglitke: yes. this can avoid the user to write the file directly.
13:39:20 <royce> I think that transaction can be gaurenteed by copy and update instead of write lines
13:39:30 <aglitke> royce: You may need to improve the locking for the FileStore class.  I would suggest a threading.Semaphore per FileStore object.
13:40:04 <aglitke> to work with the filestore you must create a session (as with ObjectStore) and that session takes the semaphore.
13:40:05 <royce> aglitke, mark suggested each for a template too
13:40:31 <aglitke> then while holding the session you can safely manipulate date.
13:40:33 <aglitke> data.
13:40:45 <aglitke> I think it needs to be global per object type.
13:40:46 <royce> got it, agitke
13:41:09 <aglitke> otherwise you could get an inconsistent list of objects if someone else is removing one.
13:41:22 <royce> right
13:41:50 <aglitke> Do we have agreement on the design?
13:41:53 <aglitke> alinefm: ?
13:42:25 <alinefm> aglitke, yeap
13:42:27 <royce> let's see if UI guys agree with this?:)
13:42:35 <aglitke> royce: agree!
13:42:47 <alinefm> we need to assign people to it
13:43:01 <aglitke> yes, that will be the next step once everyone agrees with this plan.
13:43:03 <ming> I assume the session is cherrypy session here.
13:43:10 <aglitke> ming: no
13:43:23 <aglitke> it's like ObjectStoreSession in the current code
13:43:37 <aglitke> it's a python ContextManager context
13:43:37 <shaohef> aglitke: just for the file operation session？
13:44:13 <ming> aglitke, not sure why we need that context.
13:44:28 <aglitke> It's for entering a critical section
13:44:34 <ming> It is easy to maintain a global lock across differnt cherrypy sessions.
13:44:51 <aglitke> the context is bounded by a lock acquire and release
13:45:11 <aglitke> with FIleSession(...) as ctx:
13:45:17 <aglitke> do first op
13:45:20 <aglitke> do second op
13:45:26 <aglitke> return
13:45:44 <aglitke> before entering the sub-block, the lock will be taken
13:45:56 <aglitke> after exiting the block, the lock will be released.
13:46:13 <aglitke> first op and second op are guaranteed to happen sequentially.
13:46:29 <aglitke> without any interleaving operations to corrupt data.
13:46:30 <ming> so you protect each file with one context ?
13:46:46 <royce> aglitke, it means vm/screenshot will also use file backend after that?
13:46:48 <ming> or multiple files with one context?
13:46:56 <aglitke> You can either do one global one (for all types) or one per type.
13:47:06 <aglitke> I think one per type is ok for our API right now,.
13:47:30 <royce> means templates hold a lock and vm hold a lock
13:47:37 <AdamKingIT> I don't know that we want screenshot to get more entangled with files
13:47:38 <aglitke> royce: I am ok with keeping ObjectStore for VM metadata
13:48:00 <aglitke> It does not need to be altered by external packages.
13:48:17 <ming> royce, so as to the state for the async threads.
13:48:45 <ming> s/so/the same/
13:49:15 <ming> next item?
13:49:32 <aglitke> I guess the FileStore will be simpler to implement if each file can contain only a single json object (not a list as I said before)
13:49:44 <aglitke> It means we will have a lot of files in that dir.
13:50:01 <aglitke> in /etc/kimchi/distros
13:50:12 <shaohef> aglitke:  ming: same with the sqilt3 session
13:50:31 <aglitke> I want to go down the list of tasks for this feature to make sure we are working on them all.
13:51:04 <alinefm> 1) create backend to list iso links
13:51:11 <royce> the KPI feature of alinefm
13:51:12 <ming> like { 'name': 'Fedora 19'} in one file?
13:51:19 <alinefm> 2) create UI do list iso links
13:51:24 <aglitke> * Capabilities detection
13:51:25 <aglitke> * File-based template storage
13:51:25 <aglitke> * API to list known iso templates
13:51:25 <aglitke> * UI to create a template from a known network iso
13:51:25 <aglitke> * UI to create a template from a user provided URL
13:51:38 <aglitke> oh... sorry :)
13:51:53 <alinefm> np! =) you have it ready
13:52:06 <aglitke> * Capabilities detection (alinefm) ?
13:52:16 <alinefm> yeap! =P
13:52:20 <aglitke> * File-based template storage (royce) ?
13:52:29 <aglitke> This is what we discussed above
13:52:33 <royce> right~
13:52:42 <aglitke> * API to list known iso templates (royce)?
13:52:54 <aglitke> This will read the files in /etc/kinchi/distros
13:52:57 <alinefm> this is related to iso links?
13:53:07 <ming> I think royce is overloading...
13:53:15 <xinding> * UI to create a template from a known network iso
13:53:16 <xinding> * UI to create a template from a user provided URL
13:53:16 <xinding> For these two, add the feature to redirect to edit window after creating
13:53:20 <aglitke> And will present them at the uri /config/distros
13:53:47 <shaohef> then what will I do？
13:53:49 <royce> maybe shaohef can help with this one if he wants:)
13:53:53 <aglitke> API to list known iso templates  depends on the file based storage api
13:54:12 <aglitke> sure, you guys can break it up however it works.
13:54:24 <shaohef> royce: aglitke:  yes. I can take one
13:54:35 <royce> thanks,shaohef~
13:54:38 <aglitke> xinding: You have the UI pieces covered?
13:55:15 <AdamKingIT> Can we chat a bit more about the redirect to edit after create?
13:55:17 <xinding> * UI to create a template from know isos list
13:55:24 <aglitke> maybe shaohef can give you a MockModel implementation of /config/distros that is fake so you can test the UI part
13:55:51 <shaohef> aglitke:  sure.
13:55:55 <aglitke> it is easy to mock up the backend using MockModel to enable UI testing.
13:56:18 <aglitke> shaohef: Cool.  Then when royce is ready with the backend stuff you can implement the real model.
13:56:22 <aglitke> AdamKingIT: Sure.
13:56:35 <shaohef> aglitke: yes.
13:56:36 <aglitke> I think the redirect to edit should only happen for custom URLs
13:57:02 <royce> OK, aglitke I will trying not block shaohef
13:57:02 <aglitke> Because we have no way of knowing which distro they point to.
13:57:07 <ming> editing the URLs for correction?
13:57:18 <AdamKingIT> Do we still plan to read the first few sectors of a custom URL like we do w./ local ISO?
13:57:27 <aglitke> royce: no problem.  shaohef can add a list of fake distro entries for the MockModel
13:57:33 <AdamKingIT> So we can detect distro?
13:57:39 <royce> oh! that one...
13:57:46 <aglitke> AdamKingIT: Maybe later.
13:58:08 <aglitke> You need to read 16 sectors
13:58:28 <AdamKingIT> The thing w/ edit is we do not presently allow anyone to edit the distro/version of a template, nor the icon...
13:58:35 <aglitke> oh, ok
13:58:44 <aglitke> maybe we should try to get the read working then.
13:58:56 <aglitke> Also, we can detect if the url is valid or not,
13:59:03 <AdamKingIT> So the edit would just be to edit cpu/memory (+storage pool and template when those come in)
13:59:12 <royce> I saw alinefm did change to use urlopen to validate it, maybe it can be added there
13:59:13 <alinefm> aglitke, my patch already does that
13:59:31 <ming> we already get the ISO generator patches
13:59:35 <aglitke> alinefm: Really?  Awesome!
13:59:55 <aglitke> ming: Those aren't needed for any of this,
14:00:02 <royce> alinefm validated the url , I think read sectors can be added there
14:00:24 <alinefm> yes, I mean validating the url
14:00:26 <ming> aglitke, I mean we can extend that to get the 16 first sector to know the distros.
14:00:30 <aglitke> ok... Then we pass the raw data to the same scanning code to do the checks.,
14:00:32 <royce> ming, generator is for test logic related isos
14:00:35 <shaohef> alinefm:  read 16 sectors
14:01:34 <shaohef> ming: you means read the  16 first sector  from the remote iso?
14:01:35 <aglitke> ok.  So... I think iso streaming is the largest feature by far.
14:01:40 <alinefm> right! so we need to read 16 sectors of iso streaming and use iso_info to detect?
14:01:45 <aglitke> But everyone has a piece of it.
14:01:58 <royce> alinefm, you pass it to probe_is
14:02:01 <AdamKingIT> agiltke, you going to update the wiki?
14:02:06 <royce> probe_iso
14:02:10 <aglitke> The last feature is the auth... I am making the suggested changes based on your reviews.  It should be ready to merge after that.
14:02:20 <aglitke> AdamKingIT: Sure.
14:02:31 <AdamKingIT> great...
14:02:55 <AdamKingIT> We have a few more features in flight. Not everyone is coding ISO streaming, yet anyway :-D
14:03:01 <aglitke> I don't want to consider other patches (except for bug fixes) other than these patches we have talked about.  We need to focus on these features.
14:03:15 <ming> agltike, I am enhancing some of the authentication method.
14:03:16 <royce> aglitke, the KPI one
14:03:35 <aglitke> royce: Who is working on that one?
14:03:49 <royce> alinefm, but I have some suggestion:)
14:04:00 <aglitke> ok, let's tackle it quickly.
14:04:11 <royce> I wish it could be put in single thread collected periodically
14:04:15 <AdamKingIT> aglitke: Once you get through authentication you can spend more time reviewing others
14:04:29 <aglitke> AdamKingIT: yep
14:04:41 <alinefm> royce, yes! good point
14:04:46 <ming> I would suggest aglitke put more effort on patch merging.
14:04:58 <aglitke> #action aglitke update wiki with current feature state
14:05:17 <royce> In the future we may need jurnal or history of these data which can not depend on browser access
14:05:17 <alinefm> royce, I resent the patchset yesterday. did you have a chance to see it?
14:05:19 <aglitke> ming: I can only merge when patches are ready
14:05:27 <aglitke> quality first, then merge
14:05:29 <royce> Thanks alinefm:) Haven't yet
14:05:44 <YuXin> AdamKingIT: is there any license issue of introducing jquery UI to Kimchi?
14:05:53 <royce> I'll review tomorrow, alinefm
14:06:05 <aglitke> YuXin: I do not think so.  Just include it into ui/lib
14:06:20 <YuXin> ok
14:06:21 <alinefm> royce, I've just recorded the max rate value to fill the circle properly
14:06:22 <aglitke> If it's the same license as jQuery itself then we should be ok.
14:06:24 <AdamKingIT> I don't think so either, but I forgot to discuss w/ Frank when we talked yesterday
14:06:37 <alinefm> maybe UI guys would like to change how this data is shown
14:07:33 <aglitke> royce: It's a good idea... We don't want to collect stats for each connected client.
14:07:33 <AdamKingIT> I am sure we will love it. :-) I'll check it our this afternoon
14:07:45 <AdamKingIT> out
14:07:54 <ming> alinefm, about the max rate.  Can we make the max rate dynamatically?
14:08:35 <ming> alinefm, you don't have exact the max rate of a network device.
14:08:47 <alinefm> ming, I did it
14:09:00 <ming> then, what is that?
14:09:01 <alinefm> ming, I check the max rate in each request and update it
14:09:02 <aglitke> I am not sure we want to calculate a max rate at all.  It will be misleading.  I think we should just display the actual value.
14:09:27 <alinefm> aglitke, the max rate is only to fill the circle
14:09:28 <aglitke> otherwise 1kb/s might be 100% if that's as high as it's ever been
14:09:38 <alinefm> and the center the actual value is displayed
14:09:44 <aglitke> oh, ok
14:09:46 <ming> alinefm, I will read it.
14:09:51 <aglitke> Maybe that is ok then.
14:10:09 <aglitke> ok... We have run 10 minutes over time.
14:10:18 <aglitke> Any last quick items before we end?
14:10:20 <alinefm> for example: if the max rate is 250 and the current rate is 100, the portion of circle filled will be 100/250
14:10:20 <royce> that is lovely, center actual value idea:)
14:10:34 <alinefm> but  in the center the value will be 100KB.s
14:10:42 <aglitke> alinefm: good
14:10:48 <ming> But how you know the max rate?
14:10:59 <ming> Is it static?
14:11:01 <aglitke> ming: stored in the model?
14:11:03 <royce> ming, update on every request
14:11:10 <alinefm> update in each request
14:11:41 <ming> K, let's talk off line.
14:12:07 <aglitke> ok, thanks everyone for joining!  Let's close out these features and have a great Sept release!
14:12:09 <aglitke> #endmeeting