Monday, 29 April 2013

[weSPOT] Personal informatics, workshop, chi conference and weSPOT

This weekend weSPOT (as myself on behalf of the project) attended to the workshop of personal informatics at Chi conference. It was nice how the organizers set up this workshop in a hackaton kind of way.

First we participated on the workshop madness session, a series of 2 minutes presentations where participants could introduce themselves and their work. 2 minutes is a really short period of time but enough to make the others understand what are you working on and what do you expect from the workshop. Sure! It requires pragmatism, simplicity and left aside a bit of the narcissism that characterize to every good (and not so good) researchers ;).

In fact, it was one of the issues that Mara Balestrini brought to the discussion, are personal informatics promoting narcissism? Personal informatics are pretty much about self-knowledge, but this tools also should promote empathy among the users... it is not a matter only to understand yourself, it's a matter also to understand the others. I really liked this kind of reasoning, because in our topic, learning is also important. In fact, we expect that students understand themselves through understanding their peers in the social context.

After this workshop madness session, we started our hackaton. We started to work in a project that we previously discussed through email. The members of my team were Mara Balestrini, Jon Bird, Christian Detweiler, and Mads Mærsk Frost. Basically, our team focused on how truthful are the answers when people replied to a survey due to a sociability bias. It has been a long topic discussed along the years and some people already proposed a simple solution for yes/no questions [1][2].

Jon Bird proposed to develop an app with this system. Basically the system relies on a very simple methodology, the user before answering a question has to flip a coin. If it's tail, you have to say the truth, if it's head, you have to reply yes by default. In this way, nobody knows if you has replied truthfully or not. However, statistically we know how many 'yes' we can drop from the sample and the rest are reliable 'yes'. The theory says that in this way, we can know the real percentages of the answers.

In order to demonstrate whether this system could be integrated in an app, we are going to deploy three different kind of surveys. One survey where the flipping coin methodology is not applied. Another where the user has to flip a physical coin. Finally, a third one where the user has to flip a virtual coin integrated in the system.

The ideal result would be that a social bias exists in the first one but not in the other two. But we'll see the results... we hope to deploy tomorrow during the conference.

Does someone wonder what kind of questions will be? We'll try to balance between very personal ones such as "have you ever had an affair?" and less personal ones where the social bias should be less.

We'll see what comes up from this very interesting workshop! Hope we can report something soon!

In the meantime, let's see if we can get some inspiration from this amazing conference!




Wednesday, 6 March 2013

Navi, StepUp, OpenBadges and ¿Gamification?

It was a long time ago since my last post... but is always to get back to good habits...

Yesterday there was a really nice discussion in our HCI course where we are evaluating our Open Badges approach.

In this experiment several tools take part:
  • Navi: It's the dashboard to display the badges to our students. As you probably know, we are continuously iterating our prototypes and this is not an exception ;) So feedback is welcome! Btw, this app is developed by Sven Charleer who joined our team on January.
  • StepUp back-end: If you have read previous posts on this blog, you know that I am working on trackers and visualizing this information in a meaningful way for students (or at least I try)
  • Open Badges system: We rely on Mozilla Open Badges System to give the students the possibility to share their badges with the outside world through social networks.
  • Analytics layer (sorry, it does not have any URL): The backend that contains all the rules to award the badges.
  • Activity Stream of the course: Following the same concept of TinyARM that aims to increase the awareness on what others are reading, we merge the different activity streams of the course such as twitter, blogs and badges in the activity stream, offering to filter by the different actions.
What is the goal of this experiment? Are we gamifying the course?

Badges are game elements but they are representation of achievements. Some students claimed yesterday that applying gamification to master students was a bit childish... and I am aligned with this idea... there are even current research that goes against the gamification of learning because it breaks the real motivation of learning... everybody should have their own intrinsic and extrinsic motivation for learning... however how the teacher teaches the lesson is other point... if he does it dynamically, participative, collaborative or simply boring is up to him or some rules of the institution... and usually is up to the student, to attend the f2f lessons (except if they are mandatory), to be participative, etc... and learning analytics tools can be part of these decissions.

Learning analytics are other additional resource to help the students to steer their own learning process, but is up to the student to use the tools that we provide. We usually test our applications with bachelor and master students and our assumption is that they are autonomous students... they will become engineers and computer scientists soon... so our first assumption couldn't go in different direction.

So... What are badges for us?

Our assumption is that badges are a representation of achievements and a mean to reflect on what is going in the class.

If you (as a student) are not tweeting, commenting or blogging but you see that others are getting badges for it, it may trigger a question:
  • why is the teacher giving badges to the students? The answer is clear, we are encouraging positive behavior.
There are some badges considered as neutral, but they are given periodically. Theoretically, they want to increase the awareness of what you or other student has done. We could use some chart instead, whereas badges represents an achievement, visualizations rely on the user the cognitive effort to drive conclusions and we try to simplify this reflection process.

If someone finds a fun element in this process, it's great! It will increase the motivation and it usually have positive effects! But... Learning is already fun by itself!

And what are we trying to figure out from our students? Do they consider them useful? As a reflection mean, as motivational elements, as positive feedback... they decide and:

WE LEARN FROM THEM

Wednesday, 12 September 2012

Some thoughts about yesterday symposium on awareness in technology-enhanced learning

Yesterday, Katrien did a great presentation about our most recent work. We are trying to wrap up all our case studies and to draw some generic conclusions from our experience about learning dashboards.

One of the main criticisms that we received was the kind of evaluations that we were performing with students. Typical standardized forms that we evaluate the perception of the user about the tools rather than usage data... and I agree that we cannot stop on this level... and we are doing steps ahead in this area.

We are also trying to find correlations between different kind of activities. Another criticism was about this point, if the activities are mandatory... it's normal to find correlations. And I think that it is a totally understandable assumption, but my research on this field says something different...

Trying to summarize what I am doing... I am building dashboards visualizing different kind of traces, from tweets, blogs, comments, paper reads and time spent. We deploy these dashboard in Erik's courses that they follow some kind of 'open learning' approach where we encourage students to share their results/opinions using the above mentioned social networks. The line between encouraging and being mandatory is pretty thin and for sure, there are biased results.

Based on the first assumption, if something is mandatory, you will find correlations with the grades... this is not true... for instance, commenting on each other blogs and tweeting is equally 'mandatory' (although this activity does not influence the grades) so... based on this assumption, both should correlate with the grades... sorry! There is no significance correlation in... (I let you guess which variable has no significance and try to guess why!).

We build dashboards trying to help students and teachers (I like more the idea of students... they are a bigger challenge from my point of view). How can we help them? Giving them metrics that help to understand them their performance.

I really liked the presentation of Marcus Specht yesterday, they also work quite a lot related to awareness and reflection in learning but I could also see yesterday they do it also in other interesting fields. They also get a lot of inspiration from the quantified self movement as we do. But I also think that it is a bit dangerous to simplify approaches... the quantified self movement starts from a self-knowledge or self-goal motivation... learning has a mixed of motivations... even more when a learning activity as it can be "to use a learning dashboard" is not mandatory because we think that it's optional due to a master or bachelor student should be almost autonomous in its learning decisions.

Why did I start to talk about the quantified self movement? Because understandability and motivation are linked from my point of view and conclusions from my evaluations. The use of every visualization, dashboard or tool has its own learning curve. If it's complicated, nobody will use it (also other reason because we still perform usability tests). So the metrics and the visualizations should be easily understandable.

I performed an evaluation with HCI students to compare two of my prototypes: mobile version vs big table. Some of the students that preferred the mobile version pointed out that they still wanted to have the big table available because they wanted to understand their own results. Others commented that they wanted to use exclusively the big table, because they wanted to draw conclusions by themselves.

I agree that algorithmic and computational efforts can make a big contribution to the field, I strongly believe that they are completely necessary... but we are developing tools intended to help users and users have their own feelings and opinions and if they do not like the tool, they will try to avoid to use it.

Algorithms can make a good work... but also they can be wrong... and being clustered somehow in some kind of cluster is not always nice... for instance, I am attending some courses in coursera, I am sure that people who is participative in forums, meetups and so on... they will learn much more than me... I am sure of it... If the system would give me feedback, they will categorize me probably in the group of 'slackers' and the systems totally ignores what are my priorities in my life... maybe I am person who works in a factory, twelve hours per day... and I arrive at home very tired with time enough to watch the videos and nothing else... or maybe I do not work twelve hours... but I just had recently a baby who requires most of my attention... or maybe it's true! I'm a slacker... but how do you differentiate between the three cases? I think that it is not an easy task... in the same way, that it's hard previously define what are the conclusions that your algorithm should draw from the data... another point the data... when do you start to have enough high quality dataset to draw conclusions? Coursera courses are 6 weeks long... how long have you to wait till your dataset work?

I am not claiming that our way of doing is the correct one... I really like the idea of recommenders, profiling users and so on... but maybe and only maybe... since the moment, learners are not so interested in the results of our European projects... maybe it is because we do not understand them... and for that, we do not need to do great, fancy and elaborated applications, we need to evaluate everything and to try to understand them... and my hopes are that with a little more of time... my PhD and our work can provide a bit of light in the middle of the darkness... :)


Tuesday, 5 June 2012

Invitation to participate in a survey funded by the European Commission

One colleague is conducting a survey on why some innovative SMEs do or not take part in R&D projects by the European Commission. If you hold a SME or work for one, I think that your input can be highly relevant for this study. Hopefully, the results can help to improve the funding programs becoming a useful way to provide resources to the true back-bone of the European economy.

I hope you can contribute with your highly valuable opinion. If it is your situation, please read  bellow and share it with other SMEs.

 
*Please feel free to disseminate. Apologies for any cross-postings*
We are currently running a survey on reasons why some innovative SMEs (Small to medium sized enterprises) in the ICT sector do or do not take part in R&D projects funded by the European Commission, and we would very much like to have your opinion.

By taking part in the survey, funded by the European Commission, you will not only be ensuring  that ICT SMEs will influence the Commissions project planning, but you will also be entered in to a draw to win an iPad. You can also undertake a free innovation audit for your company.

If you are interested in more information about this study, please visit our website: www.smenonparticipation.eu


To take part in the survey you should be either an innovative ICT SME or an ICT Association from the EU-27 or associated countries (including Switzerland, Israel, Norway, Iceland, Liechtenstein, Turkey, Croatia, Macedonia, Serbia, Albania, Montenegro, Bosnia & Herzegovina, Faroe Islands and Moldova).

The survey should take no more than 15 minutes to complete and does not require any prior knowledge of R&D funding programmes. It is currently available in English, and will soon be available in French and German as well. If you have any questions please get in touch directly by email (noaa.barak@theia.eu) or by filling in this contact form.

Our target is to collect the views of as many companies as possible so that our findings truly represent the views of companies in your sector. We would appreciate it if you could pass this invitation forward to your contacts, if you feel it could be of interest to them.

Thank you in advance for your support, and good luck in the draw!
 

Friday, 25 May 2012

New prototype and playing with quartiles and outliers in STEP UP!

"subliminal" advertisement - REMEMBER: We organize LAK'13, are you ready for your submission? Do you have any good, amazing and original idea? Come on! Let's do it! ;) - the "subliminal" advertisement is finished- yes... I know... I don't really understand the concept of subliminal :)

As I already presented at the end of my presentation at LAK'12, we are trying to simplify STEP UP!.

First, we did an small prototype. You already know our iteration process methodology, aren't you? Otherwise, read one of our papers! ;)



 We didn't evaluate this prototype because I had a PhD meeting with Katrien and Erik, and during the discussion came up the idea of developing it for mobile devices. So then, I moved the code to JQuery mobile, giving us the following result:

What did we change from one prototype to other?

Basically, both follow the same concept except for the colors. In the first prototype, we used:
  • red: Bad student!
  • yellow: Careful! Maybe you should work more, shouldn't you?
  • green: Good boy/girl! Good student!
But we realized that we can not say that from the activity of a social network, at least, with the analysis that we do. We try to encourage students to reflect on their data, we do not intend to say: "You are a good/bad student". So we decided to give different meaning to the colors:
  • blue: cold activity. Dude! Your activity is lower than your peers. Up to you! Maybe you don't need anything from the community, but maybe the community need something from you. We are also learning how we can become good open learning students. Share your learning and knowledge for free and you will receive something back... not sure why, when and how but do it and you will see that!
  • green: you are in the average activity... you are participating as most of your peers. It does not mean that your contributions are good, but you have at least the habit to contribute to the community, and it is also part of the process.
  • red: "You are in the hot zone". What is going on with you? Are you a social network addicted? Are you addicted to study? Go to the real life and enjoy a beer with your peers! Just kidding.. for sure... this is not the message that we want to send to the students, on the other hand, it is quite similar... the student is participating over the average activity. Is it really necessary? If others are not so active, why are you so actively contributing? Reflect on that! If you really need it, do it! But it's important to be aware of this aspect.
Is the prototype already plugged to real data?

Yes! It is! We have already started to play with student data. For instance, the screenshot above contains real data of this week. We have already finished the lectures of the course so this week there is no so many activity.

Once that we had the prototype, we had to decide what would be the criteria to translate data activity to a percentage to fill the bar. First, we thought in the arithmetic mean, we implement it. But we were not totally convinced... why? How do you detect outliers?

So we decided to go for the concept of box plots calculating quartiles and outliers. We found a really easy way to calculate quartiles that made our work easier. And how can we calculate the outliers? Also it's very simple.

  • IQR = Q3 - Q1
  • Up Outliers > Q3 + 1.5*IQR
  • Down Outliers < Q1 - 1.5*IQR
 All the students with an activity between Q3 and Q1 have filled their bar with color and the percentage is between 25% and 75%.

Students with an activity bellow Q1 go from 0 to 25% (and blue color) and above Q3 (and red color), from 75 to 100%. Outliers are assigned respectively to 0% and 100%.

What do you think? Does it make sense? If you have better idea, don't hesitate to share it with us!
  

Wednesday, 9 May 2012

LAK'12 conclusions

After some days, I've had time to think about LAK'12 conference.

BTW, really important note: Next year, we are going to organize LAK'13 in Leuven! So prepare your submissions... we are waiting for your contribution!

But how do we see Learning Analytics? You can check some slides from my Prof. Erik Duval or some of his thoughts about Educational Data Mining vs Learning Analytics.

But, what are my conclusions after LAK'12?

I would summarize it with a really nice photo that I took during my days off in Vancouver.


What does this photo mean to me?

The conference is finished, but now we have a nice picture of what our community in learning analytics is working on. The mountains (our goals) are still far away, but we only have to swim (to work) to get there. One day is over, but we are sure that tomorrow is going to start another great day. I think that it's really nice when you finish a conference with this kind of feeling.

It was a really nice experience that give me a lot of food for thought. I would highlight a really nice talk with Jon Dron who during the demo session gave me some interesting pointers (i.e. The Design of Everyday Things and Dr. Vive Kumar (still I have to take a look to his work)) and I had the opportunity to discuss with him issues regarding privacy and my PhD topic.

Another positive aspect is to read a bit the conclusions from others (i.e Abelardo's blog, Doug Clow's Blog, Audrey Watters' blog, ... ). People who share their thoughts about the topic... the common conclusion after every conference is that sharing knowledge is the key to progress in a research field and there is people doing a great job in that! So I only can say thanks guys to share your thought with all of us!

After my presentation, I had also a really nice talk about how to engage people in the reflection process. One easy argument is to include this process in the learning itinerary. However, we are often tracking sensible data and we can not force students to give it (i.e. tracking data beyond of the LMS common problem with Abelardo's group work). In our case, we track data from different systems and they have to give us their API keys to have access to their own data. If we include the reflection process in the learning itinerary, the tracking becomes mandatory... and somehow the final feeling is that everything is corrupted (in addition it can be against the law). We are trying to engage students in the process of open learning, show and share your thoughts with the world and somehow it will come back with some additional food for thought. Sharing your information and reflection should be a voluntary and participatory process. It is our premise.

Also, I had a really nice talk with David García-Solórzano from the Open University of Catalonia regarding his paper: "Educational Monitoring Tool Based on Faceted Browsing and Data Portraits". Damn! They are really doing nice work. I really encouraged him to evaluate their prototype because I think that it has a lot of possibilities. They are concentrating a lot of information and I think that they need to get some feedback now. Also, it was really nice to hear that he got a lot of inspiration from the paper "Attention please! Learning analytics for visualization and recommendation" by Erik Duval. You feel somehow lucky and think: "Yeah! And I am doing the PhD with him". Afterwards, you wonder why he hasn't still fired you after questioning all his ideas in the PhD meetings... I guess that it's the PhD student syndrome, we think that we know more than we actually know... or maybe it's my personal syndrome but I feel better thinking that others share the same problem... as the saying goes:

"It is a fool's consolation the think everyone is in the same boat"

Cheers! ;)

Wednesday, 25 April 2012

Preparing LAK'12 presentation!

I was thinking to write some lines about my presentation in LAK'12 conference. This presentation is about the paper Goal-oriented visualizations of activity tracking: a case study with engineering students and just now taking a look to the program, I saw that it will be broadcasted by video streaming. It is getting funnier, before I was just worried about giving a presentation with approximately one hundred attendees... now it will be broadcasted... so... more people... even more, I guess that it will be recorded so... more fun? :-) anyway... summarizing... it is a bit scary, isn't it? I think that I'll never get used to present my work, although it's always a nice learning experience (once finished :-))

However thinking about it, my conclusion is that I don't have anything to worry about... because I have a super presentation thanks to the feedback got from my colleagues in one internal try-out presentation.

I would like to share it with you and get maybe some additional feedback? I am still not convinced about mixing bar charts and box plots to show similar information, however the SUS questionnaire is not represented using box plots because the result was a bit weird. Anyway any feedback is welcome.

So... here you go!