PhD finished - lessons learned

I defended successfully my PhD on 20th of November. As you can guess, it was a relief. Almost 6 years working on a book that synthesises all the work done... at that moment, you think how many experiments were not successfully published but were good ideas yet.

That was a very long journey.

I could write a long text describing the positive and negative things of this long journey - you know reflecting on my journey... believed me... I have done it several times... so I will summarise them in just three points:

1. Keep working and just do the best you can... and even when you do that, be ready for criticism. Think that when you read a published paper from other PhD student:

• the promoter agreed with the content. In many cases, even co-promoters and other colleagues from the same and other institutions have agreed with the content;
• three reviewers agreed that the content is worth to be published;
• editors or program chairs have given the ok...

       So, if you think this was going to be easy, give up and change your career... 

2. probably one of the biggest lessons learned from your PhD will be: learning how to deal with disagreement. Good ideas are not enough, the idea and the results will require the consensus of at least 4 people with different background, expertise and interests. 

3. don't set up wrong goals (maybe it's a too hard statement)... the MIT has a limited number of places ;). Don't get me wrong, there are people who need this sort of goals. However, the PhD is about the journey and it will take you between 3 and 5 years. Your most important goal is to improve. It's an individual goal. Remember that each publication will be reviewed by 4 people at least (see point 2 ;)). Do you think that if you finish your thesis with 5 papers and your thesis that probably will involve and average of 15 and 20 people who thought that your work is relevant for the field, aren't you a suitable candidate for whatever institution around the world? What if, in addition, you won best paper awards, participated organising workshops, reviewed papers and created an interesting network of contacts?

Don't you think that it will keep yourself busy for 3-5 years? Good work will lead you to nice, good and wonderful learning experiences and do not worry about the next step. If you nailed your PhD, many interesting groups will want and need you.

Summarising: keep up the good work fellows!

I must admit that I struggled with the three lessons. But don't get them wrong... it doesn't mean that you need to give up on your beliefs... on the contrary... believe on them hardly.... they will keep you alive...

Commit to your ideas! Keep your mind open and be aware that science is team work more than you think! 

In my case, I moved back to the private sector in August as you can check on my LinkedIn and I am quite happy with the decision. Btw, we are looking for a Senior Big Data Engineer, so feel free to spread the word. I am sure that in this company, we know where we start but we don't know where we are going to end because we are going to face very interesting challenges quite soon. I would love to partner efforts and collaborate with a Big Data Engineer. 

Now thinking what is the best platform to blog... probably Medium but thinking also to blog on LinkedIn... but this is my last blog post here: new topics and new blog posts are about to come! (somewhere else ;)) 

I also share the slides of my PhD presentation if you want to check them out!

Ps: As I said, this PhD was a team effort. And I must thank especially to my promoters Katrien Verbert and Erik Duval who is dealing a very particular fight! Good luck!

Public PhD defense from Jose Luis Santos Odriozola

The weSPOT meeting is over... now back home!

The weSPOT meeting is over! Nice project, nice people and splendid food in a nice city: Graz :)

Six deliverables are in the oven and close to see the light of the real world ;).

KU Leuven is in charge of D3.3: User management and badges system. And I personally like the content.

The deliverable starts with the explanation of the weSPOT OAuth provider. We had a problem in weSPOT. Our log in system relies on OAuth providers... why? Simple... we wanted to simplify the process of users enrolling into our inquiry environment. So users could join our system using their Facebook and Google accounts. However, kids bellow thirteen shouldn't have the accounts... so we needed to provide them another mechanism to sign up in our system.

weSPOT created its own OAuth provider in the cloud. The provider is hosted in Google App Engine.

If you really want to know more about it... just check it out in our site! And if you want to deploy yours... don't hesitate to contact us!

We have also created several badges to engage users in the use of the system. Here you can see a screenshot.

Besides the interface and the rules, we also created a some sort of Open Badges API that provides the basic functionality to create, award and store the badges. If you consider it convenient, you can check the API. The source code is available, just in case you want to deploy your own instance ;).

Almost forgot! We also implement and will offer recommendation services. But this will have to wait... we don't still have them in the production version... be patient! :)

weSPOT attends #OBIE2014 and #icwl2014

We did it again! weSPOT attended two great events such as ICWL and OBIE (The 1st International Workshop on Open Badges in Education).

It was a good opportunity to show all our designs, thoughts and experiences with Open Badges.

We had discussions about questions such as:

How many badges should we design for a course?

What are the first actions that someone should take when s/he decides to deploy badges in her/his course?

Shall badges be a representation of competence/skills?

Moreover, Nate Otto presented a very interesting project that it is worth to take a look. This project discusses about design principles for badges. 

I would like to highlight also one of the papers presented in the main conference: Open Badges: Challenges and Opportunities . Authors reflect about the actual possibilities, limitations and future work in the field. They were also some of the organizers of the Workshop.

We have still many open questions since the concept of badges and its acceptance is still a challenge.

We also discussed about the perception that students have when we introduced badges. I explained our experience when some students were not very positive towards badges since, from their point of view, they consider Learning as a very serious activity. We also addressed the problem in an informal way explaining them that we look at them as goal representations.

One of the main conclusions that the badges acceptance require dialogue with all the different stakeholders.

Badges as we talk in our papers have different aspects to consider. They are considered game elements, but also represent goals and they have social recognition. We are interested in these two last elements. However, we need to deal with the possible perception of the first element.

Many experiments to deploy and fun experiences to enjoy... and soon also in weSPOT :).

Improving teacher awareness through activity, badge and content from José Luis Santos

weSPOT attends LAK and OLA meeting - Is learning Analytics just a big hype?

Finally, we just get to the end of an exhausting but amazing week! Many meetings, talks and the annoying jet lag!

 LAK and OLA has been an amazing opportunity to attend to many presentations and to have exhilarating conversations with many people... so let's try to summarize the experience a bit.

Personally, I had the opportunity to meet some folks from OUNL (Maren and Hendrick), that are working around Learning Analytics. They are  involved in the LACE project. One of the goals of LACE is to build a framework of quality indicators for learning analytics. They are in the brainstorming phase trying to collect those indicators, and weSPOT has contributed.

Hendrick is also in the process of collecting real data from their institutional LMS, what can bring them the opportunity to run different tests on such amazing dataset.

We had the opportunity to talk with folks from the Apereo foundation  as well, concretely with Alan Berg and Sandeep Jayaprakash. They are contributing to the OAAI initiative and trying to define the flow of information in a Learning Analytics Systems.

Both collaborations can be an opportunity to contribute with the expertise of KU Leuven: Learning Dashboards. Therefore, Sven will have soon the opportunity to do more of his cool stuff with more and different data.

Along the weekend, we participated in the the Open Learning Analytics meeting, somehow we are trying to define the roadmap for SOLAR and exploring how we can collaborate with each other, funding possibilities, etc.

But... what about the conference itself?

 First, I will share the proceedings and my slides (I have to admit though that the last slide was the one who got more attention :-P). Also it is worth mentioning the amazing work that some of the attendees did reporting on their own blogs, however, I think that Doug Clow and Stian Håklev were clearly the best.

 
(Advertising spot: Btw, Stian is about to finish his PhD as well as I (I hope ;)) and we are exploring the possibilities for "the next step"... yep... that step that every PhD student is scare of... what to do next???? Anyway... don't hesitate to contact us if you are looking for some collaboration! ;))

  But again... what is my opinion about the conference?

  Many cool things and impressive analyses of the data... but I have to mention two down sides:

• There were no dashboard presentations this year.
• I have the feeling that many of the learning analytics folks forget about the HCI aspects of learning analytics.

  Several times I have heard from many people the sentence of: "We are trying to solve a problem that may be does not exist, but addressing this nonexistent problem, we generate another problem that maybe we can address".

  But... what is going on? Why does people have this question in mind?

  They do cool stuff... but the adoption of their artifacts is slow or nonexistent.  Therefore... they come up with apparently a logical conclusion: I am doing something cool and useful, but if people don't use it... it is because they may not have the need.

  At one of the dinners, Abelardo Pardo was explaining one of his workshops experiences in Australia and New Zealand with academics... and one of the teachers asked just at the beginning of the workshop: what is wrong with the traditional way of lecturing?

  But we know that alternative ways of teaching influence certain aspects such as motivation, attention and novelty, and they may influence positively to the individual and social learning process.

  Is there a need? Ok... probably if we go to the Aristotle's definition of 'necessity'... there is no need... but we can get better on what we are doing and this first assumption is what should trigger the use of alternative methods... the goal of becoming better... although I have to admit that probably not everybody pursue this goal.

  However, HCI does quite a lot of research in this field of technology adoption. What are the common problems to attract user attention, how we can capture the user attention, how we can engage users in this process... and so on... and this kind of studies is something that I miss in LAK...

  We'll see if all these talks, events and experiences end up in some fruitful collaboration that it is what really matters... in the end... the knowledge acquisition is already done... so let's try to transfer some knowledge now...

Ah! and don't forget to watch the TED Talk about our view in Open Learning Analytics!

Reveal-it applied in an educational context!

As I mentioned in my previous post, we (Erik, Gonzalo and myself) attended the Chi conference. More concretely, we attended one session about "Tensions in Social Media" due to there was a very interesting paper called "Reveal-it!: The Impact of a Social Visualization Projection on Public Awareness and Discourse" and the fact that the authors of the paper were from my former university (UPF) and one of the co-authors was my assessor Andrew Vande Moere increased my interest to know more about it.

What was Reveal-it about?

Source of the image: blog of the primary author on the paper

I would strongly recommend you to read it. I ask in advance apologies to the authors whether I forget some important details about the paper, but summarizing this paper describes a set of experiments where energy expenses are visualized on an ambient display in different public spaces . The visualization is designed in order to increase the user awareness of energy consumption. The experiment also digs into how these public visualizations can trigger social discourse. Users that pass by the visualization can report their expenses and their information is added to the visualization. Such information is highlighted in order to attract the attention of the user and to trigger reflection. The information of the user can be compared with the expenses average of her/his neighborhood.

What did we do?

We wanted to test this application in our context, education. So we started to think about how we could apply a similar concept to our students. It could also be an alternative to our big table overview.

We wanted to test the application with our students who currently use StepUp! (a bit of explanation  here), Navi (a bit more here) and the activity stream (that aggregates all the activity of the course). So this evaluation could help us to understand how Reveal-it could complement our current work.

Ok, so... what do we have? We have students and we track different activity, tweets, blogs, time, badges... they can work in groups or individually. We wanted to test it with our current students and  the courses are ending... so they do not have to report new activity to the system... but still we wanted some interaction with the visualization with a second device such as highlighting the user activity...

Our first approach was to do the analogy between neighborhoods and groups. But we found two main issues:

•  We had to create one visualization per activity. For instance, for our #chikul13 students, we created four different visualizations (per blog posts, blog comments, tweets and earned badges along the course). Reveal-it aggregates gas and electricity expenses because both are paid with the same currency, but the nature of our data is completely different and to find a common way to measure it was difficult.
• Most of our #thesis12 students do not work in groups, so the analogy of neighborhood and groups did not work for this case study.

 We thought that to keep all the data on one unique visualization made more sense. So we forgot about the analogy and we went for a simpler approach. Neighborhoods are different activities such as tweeting, blogging, commenting, spending time and earning badges. And each user should be represented in every "neighborhood". In this way, the students can see in a glance how their efforts are distributed compared with the others and the mean.

What was the result?


So after Sam and I tweaked the code, this was the result for our #thesis12 and #chikul13 students:

And the students could highlight their usernames using a simple mobile web app:

What did we do?

We evaluated the tool during a poster demo session of our #thesis12 students where also our #chikul13 students were invited to participate. In fact, we did two simultaneously evaluations. Sven that focuses on how we can enhance collaborative reflection with multi-touch and big devices evaluated his tabletop app.

We projected the two visualizations, each one in different walls.

What were the first reactions from the students? A couple of #thesis12 students asked me before the start of the poster session if it was a #chikul13 project or something like that. I told them that the visualization was displaying their data, one stayed for the evaluation, the other almost ran away...

After this, I stayed a bit away of the visualization and I didn't see any student who paid a lot of attention to the visualizations. Consequently they didn't even read the text where was explained that they could interact with the visualization. So before losing the opportunity, I started to ask some students if they could evaluate the visualization.

I introduced them the tool explaining that it relies on a concept of public and ambient displays but that it enables interaction through a web mobile app. I let them use the app with my own mobile and they started highlighting their username and afterwards others usernames.

I highlight some findings got from the interviews (10 interviews):

•  Two groups of two people understood faster the visualization and they had some fun (at least they laughed) while they were comparing with each other.
• All the individuals needed a bit of help to understand completely the visualization.
• Three person understood the bars as a chronological representation of their own activity. Each bar was a week instead of a user. It can be a bias because so far StepUp! and Navi they use weeks as a granularity level to represent the data.
• But the most important perception, at least from my point of view, is that most of them expect to interact with the visualization, for instance, in order to highlight the outlier users. However, when I inquiry them about if they think that was a required feature, they replied that maybe to compare themselves with the mean was already ok.
• Also I asked them what would they rather prefer, whether the big table visualization or Reveal-it. The opinion was quite divided because the big table overview gives them more information, however everybody agreed that it was a fancier and nicer visualization that enables you to get a quick status of your activity compared with the others.

Still we have to research a bit more about it. But Reveal-it is considered more or less equally useful than StepUp! and Navi. On the other hand the #thesis12 students would feel less confident with such public visualizations used in public spaces than #chikul13 students. And they do not find that interacting with the visualization with a second device (in this case a web mobile app makes a lot of sense).

[weSPOT] Personal informatics, workshop, chi conference and weSPOT

This weekend weSPOT (as myself on behalf of the project) attended to the workshop of personal informatics at Chi conference. It was nice how the organizers set up this workshop in a hackaton kind of way.

First we participated on the workshop madness session, a series of 2 minutes presentations where participants could introduce themselves and their work. 2 minutes is a really short period of time but enough to make the others understand what are you working on and what do you expect from the workshop. Sure! It requires pragmatism, simplicity and left aside a bit of the narcissism that characterize to every good (and not so good) researchers ;).

In fact, it was one of the issues that Mara Balestrini brought to the discussion, are personal informatics promoting narcissism? Personal informatics are pretty much about self-knowledge, but this tools also should promote empathy among the users... it is not a matter only to understand yourself, it's a matter also to understand the others. I really liked this kind of reasoning, because in our topic, learning is also important. In fact, we expect that students understand themselves through understanding their peers in the social context.

After this workshop madness session, we started our hackaton. We started to work in a project that we previously discussed through email. The members of my team were Mara Balestrini, Jon Bird, Christian Detweiler, and Mads Mærsk Frost. Basically, our team focused on how truthful are the answers when people replied to a survey due to a sociability bias. It has been a long topic discussed along the years and some people already proposed a simple solution for yes/no questions [1][2].

Jon Bird proposed to develop an app with this system. Basically the system relies on a very simple methodology, the user before answering a question has to flip a coin. If it's tail, you have to say the truth, if it's head, you have to reply yes by default. In this way, nobody knows if you has replied truthfully or not. However, statistically we know how many 'yes' we can drop from the sample and the rest are reliable 'yes'. The theory says that in this way, we can know the real percentages of the answers.

In order to demonstrate whether this system could be integrated in an app, we are going to deploy three different kind of surveys. One survey where the flipping coin methodology is not applied. Another where the user has to flip a physical coin. Finally, a third one where the user has to flip a virtual coin integrated in the system.

The ideal result would be that a social bias exists in the first one but not in the other two. But we'll see the results... we hope to deploy tomorrow during the conference.

Does someone wonder what kind of questions will be? We'll try to balance between very personal ones such as "have you ever had an affair?" and less personal ones where the social bias should be less.

We'll see what comes up from this very interesting workshop! Hope we can report something soon!

In the meantime, let's see if we can get some inspiration from this amazing conference!

Navi, StepUp, OpenBadges and ¿Gamification?

It was a long time ago since my last post... but is always to get back to good habits...

Yesterday there was a really nice discussion in our HCI course where we are evaluating our Open Badges approach.

In this experiment several tools take part:

• Navi: It's the dashboard to display the badges to our students. As you probably know, we are continuously iterating our prototypes and this is not an exception ;) So feedback is welcome! Btw, this app is developed by Sven Charleer who joined our team on January.
• StepUp back-end: If you have read previous posts on this blog, you know that I am working on trackers and visualizing this information in a meaningful way for students (or at least I try)
• Open Badges system: We rely on Mozilla Open Badges System to give the students the possibility to share their badges with the outside world through social networks.
• Analytics layer (sorry, it does not have any URL): The backend that contains all the rules to award the badges.
• Activity Stream of the course: Following the same concept of TinyARM that aims to increase the awareness on what others are reading, we merge the different activity streams of the course such as twitter, blogs and badges in the activity stream, offering to filter by the different actions.

What is the goal of this experiment? Are we gamifying the course?

Badges are game elements but they are representation of achievements. Some students claimed yesterday that applying gamification to master students was a bit childish... and I am aligned with this idea... there are even current research that goes against the gamification of learning because it breaks the real motivation of learning... everybody should have their own intrinsic and extrinsic motivation for learning... however how the teacher teaches the lesson is other point... if he does it dynamically, participative, collaborative or simply boring is up to him or some rules of the institution... and usually is up to the student, to attend the f2f lessons (except if they are mandatory), to be participative, etc... and learning analytics tools can be part of these decissions.

Learning analytics are other additional resource to help the students to steer their own learning process, but is up to the student to use the tools that we provide. We usually test our applications with bachelor and master students and our assumption is that they are autonomous students... they will become engineers and computer scientists soon... so our first assumption couldn't go in different direction.

So... What are badges for us?

Our assumption is that badges are a representation of achievements and a mean to reflect on what is going in the class.

If you (as a student) are not tweeting, commenting or blogging but you see that others are getting badges for it, it may trigger a question:

• why is the teacher giving badges to the students? The answer is clear, we are encouraging positive behavior.

There are some badges considered as neutral, but they are given periodically. Theoretically, they want to increase the awareness of what you or other student has done. We could use some chart instead, whereas badges represents an achievement, visualizations rely on the user the cognitive effort to drive conclusions and we try to simplify this reflection process.

If someone finds a fun element in this process, it's great! It will increase the motivation and it usually have positive effects! But... Learning is already fun by itself!

And what are we trying to figure out from our students? Do they consider them useful? As a reflection mean, as motivational elements, as positive feedback... they decide and:

WE LEARN FROM THEM

Some thoughts about yesterday symposium on awareness in technology-enhanced learning

Yesterday, Katrien did a great presentation about our most recent work. We are trying to wrap up all our case studies and to draw some generic conclusions from our experience about learning dashboards.

One of the main criticisms that we received was the kind of evaluations that we were performing with students. Typical standardized forms that we evaluate the perception of the user about the tools rather than usage data... and I agree that we cannot stop on this level... and we are doing steps ahead in this area.

We are also trying to find correlations between different kind of activities. Another criticism was about this point, if the activities are mandatory... it's normal to find correlations. And I think that it is a totally understandable assumption, but my research on this field says something different...

Trying to summarize what I am doing... I am building dashboards visualizing different kind of traces, from tweets, blogs, comments, paper reads and time spent. We deploy these dashboard in Erik's courses that they follow some kind of 'open learning' approach where we encourage students to share their results/opinions using the above mentioned social networks. The line between encouraging and being mandatory is pretty thin and for sure, there are biased results.

Based on the first assumption, if something is mandatory, you will find correlations with the grades... this is not true... for instance, commenting on each other blogs and tweeting is equally 'mandatory' (although this activity does not influence the grades) so... based on this assumption, both should correlate with the grades... sorry! There is no significance correlation in... (I let you guess which variable has no significance and try to guess why!).

We build dashboards trying to help students and teachers (I like more the idea of students... they are a bigger challenge from my point of view). How can we help them? Giving them metrics that help to understand them their performance.

I really liked the presentation of Marcus Specht yesterday, they also work quite a lot related to awareness and reflection in learning but I could also see yesterday they do it also in other interesting fields. They also get a lot of inspiration from the quantified self movement as we do. But I also think that it is a bit dangerous to simplify approaches... the quantified self movement starts from a self-knowledge or self-goal motivation... learning has a mixed of motivations... even more when a learning activity as it can be "to use a learning dashboard" is not mandatory because we think that it's optional due to a master or bachelor student should be almost autonomous in its learning decisions.

Why did I start to talk about the quantified self movement? Because understandability and motivation are linked from my point of view and conclusions from my evaluations. The use of every visualization, dashboard or tool has its own learning curve. If it's complicated, nobody will use it (also other reason because we still perform usability tests). So the metrics and the visualizations should be easily understandable.

I performed an evaluation with HCI students to compare two of my prototypes: mobile version vs big table. Some of the students that preferred the mobile version pointed out that they still wanted to have the big table available because they wanted to understand their own results. Others commented that they wanted to use exclusively the big table, because they wanted to draw conclusions by themselves.

I agree that algorithmic and computational efforts can make a big contribution to the field, I strongly believe that they are completely necessary... but we are developing tools intended to help users and users have their own feelings and opinions and if they do not like the tool, they will try to avoid to use it.

Algorithms can make a good work... but also they can be wrong... and being clustered somehow in some kind of cluster is not always nice... for instance, I am attending some courses in coursera, I am sure that people who is participative in forums, meetups and so on... they will learn much more than me... I am sure of it... If the system would give me feedback, they will categorize me probably in the group of 'slackers' and the systems totally ignores what are my priorities in my life... maybe I am person who works in a factory, twelve hours per day... and I arrive at home very tired with time enough to watch the videos and nothing else... or maybe I do not work twelve hours... but I just had recently a baby who requires most of my attention... or maybe it's true! I'm a slacker... but how do you differentiate between the three cases? I think that it is not an easy task... in the same way, that it's hard previously define what are the conclusions that your algorithm should draw from the data... another point the data... when do you start to have enough high quality dataset to draw conclusions? Coursera courses are 6 weeks long... how long have you to wait till your dataset work?

I am not claiming that our way of doing is the correct one... I really like the idea of recommenders, profiling users and so on... but maybe and only maybe... since the moment, learners are not so interested in the results of our European projects... maybe it is because we do not understand them... and for that, we do not need to do great, fancy and elaborated applications, we need to evaluate everything and to try to understand them... and my hopes are that with a little more of time... my PhD and our work can provide a bit of light in the middle of the darkness... :)

Invitation to participate in a survey funded by the European Commission

One colleague is conducting a survey on why some innovative SMEs do or not take part in R&D projects by the European Commission. If you hold a SME or work for one, I think that your input can be highly relevant for this study. Hopefully, the results can help to improve the funding programs becoming a useful way to provide resources to the true back-bone of the European economy.

I hope you can contribute with your highly valuable opinion. If it is your situation, please read  bellow and share it with other SMEs.
 

*Please feel free to disseminate. Apologies for any cross-postings*

We are currently running a survey on reasons why some innovative SMEs (Small to medium sized enterprises) in the ICT sector do or do not take part in R&D projects funded by the European Commission, and we would very much like to have your opinion.

By taking part in the survey, funded by the European Commission, you will not only be ensuring  that ICT SMEs will influence the Commissions project planning, but you will also be entered in to a draw to win an iPad. You can also undertake a free innovation audit for your company.

Click here to directly enter the survey and draw for an iPad

If you are interested in more information about this study, please visit our website: www.smenonparticipation.eu

To take part in the survey you should be either an innovative ICT SME or an ICT Association from the EU-27 or associated countries (including Switzerland, Israel, Norway, Iceland, Liechtenstein, Turkey, Croatia, Macedonia, Serbia, Albania, Montenegro, Bosnia & Herzegovina, Faroe Islands and Moldova).

The survey should take no more than 15 minutes to complete and does not require any prior knowledge of R&D funding programmes. It is currently available in English, and will soon be available in French and German as well. If you have any questions please get in touch directly by email (noaa.barak@theia.eu) or by filling in this contact form.

Our target is to collect the views of as many companies as possible so that our findings truly represent the views of companies in your sector. We would appreciate it if you could pass this invitation forward to your contacts, if you feel it could be of interest to them.

Thank you in advance for your support, and good luck in the draw!

 

New prototype and playing with quartiles and outliers in STEP UP!

"subliminal" advertisement - REMEMBER: We organize LAK'13, are you ready for your submission? Do you have any good, amazing and original idea? Come on! Let's do it! ;) - the "subliminal" advertisement is finished- yes... I know... I don't really understand the concept of subliminal :)

As I already presented at the end of my presentation at LAK'12, we are trying to simplify STEP UP!.

First, we did an small prototype. You already know our iteration process methodology, aren't you? Otherwise, read one of our papers! ;)

 We didn't evaluate this prototype because I had a PhD meeting with Katrien and Erik, and during the discussion came up the idea of developing it for mobile devices. So then, I moved the code to JQuery mobile, giving us the following result:

What did we change from one prototype to other?

Basically, both follow the same concept except for the colors. In the first prototype, we used:

• red: Bad student!
• yellow: Careful! Maybe you should work more, shouldn't you?
• green: Good boy/girl! Good student!

But we realized that we can not say that from the activity of a social network, at least, with the analysis that we do. We try to encourage students to reflect on their data, we do not intend to say: "You are a good/bad student". So we decided to give different meaning to the colors:

• blue: cold activity. Dude! Your activity is lower than your peers. Up to you! Maybe you don't need anything from the community, but maybe the community need something from you. We are also learning how we can become good open learning students. Share your learning and knowledge for free and you will receive something back... not sure why, when and how but do it and you will see that!
• green: you are in the average activity... you are participating as most of your peers. It does not mean that your contributions are good, but you have at least the habit to contribute to the community, and it is also part of the process.
• red: "You are in the hot zone". What is going on with you? Are you a social network addicted? Are you addicted to study? Go to the real life and enjoy a beer with your peers! Just kidding.. for sure... this is not the message that we want to send to the students, on the other hand, it is quite similar... the student is participating over the average activity. Is it really necessary? If others are not so active, why are you so actively contributing? Reflect on that! If you really need it, do it! But it's important to be aware of this aspect.

Is the prototype already plugged to real data?

Yes! It is! We have already started to play with student data. For instance, the screenshot above contains real data of this week. We have already finished the lectures of the course so this week there is no so many activity.

Once that we had the prototype, we had to decide what would be the criteria to translate data activity to a percentage to fill the bar. First, we thought in the arithmetic mean, we implement it. But we were not totally convinced... why? How do you detect outliers?

So we decided to go for the concept of box plots calculating quartiles and outliers. We found a really easy way to calculate quartiles that made our work easier. And how can we calculate the outliers? Also it's very simple.

• IQR = Q3 - Q1
• Up Outliers > Q3 + 1.5*IQR
• Down Outliers < Q1 - 1.5*IQR

 All the students with an activity between Q3 and Q1 have filled their bar with color and the percentage is between 25% and 75%.

Students with an activity bellow Q1 go from 0 to 25% (and blue color) and above Q3 (and red color), from 75 to 100%. Outliers are assigned respectively to 0% and 100%.

What do you think? Does it make sense? If you have better idea, don't hesitate to share it with us!
  

LAK'12 conclusions

After some days, I've had time to think about LAK'12 conference.

BTW, really important note: Next year, we are going to organize LAK'13 in Leuven! So prepare your submissions... we are waiting for your contribution!

But how do we see Learning Analytics? You can check some slides from my Prof. Erik Duval or some of his thoughts about Educational Data Mining vs Learning Analytics.

But, what are my conclusions after LAK'12?

I would summarize it with a really nice photo that I took during my days off in Vancouver.

What does this photo mean to me?

The conference is finished, but now we have a nice picture of what our community in learning analytics is working on. The mountains (our goals) are still far away, but we only have to swim (to work) to get there. One day is over, but we are sure that tomorrow is going to start another great day. I think that it's really nice when you finish a conference with this kind of feeling.

It was a really nice experience that give me a lot of food for thought. I would highlight a really nice talk with Jon Dron who during the demo session gave me some interesting pointers (i.e. The Design of Everyday Things and Dr. Vive Kumar (still I have to take a look to his work)) and I had the opportunity to discuss with him issues regarding privacy and my PhD topic.

Another positive aspect is to read a bit the conclusions from others (i.e Abelardo's blog, Doug Clow's Blog, Audrey Watters' blog, ... ). People who share their thoughts about the topic... the common conclusion after every conference is that sharing knowledge is the key to progress in a research field and there is people doing a great job in that! So I only can say thanks guys to share your thought with all of us!

After my presentation, I had also a really nice talk about how to engage people in the reflection process. One easy argument is to include this process in the learning itinerary. However, we are often tracking sensible data and we can not force students to give it (i.e. tracking data beyond of the LMS common problem with Abelardo's group work). In our case, we track data from different systems and they have to give us their API keys to have access to their own data. If we include the reflection process in the learning itinerary, the tracking becomes mandatory... and somehow the final feeling is that everything is corrupted (in addition it can be against the law). We are trying to engage students in the process of open learning, show and share your thoughts with the world and somehow it will come back with some additional food for thought. Sharing your information and reflection should be a voluntary and participatory process. It is our premise.

Also, I had a really nice talk with David García-Solórzano from the Open University of Catalonia regarding his paper: "Educational Monitoring Tool Based on Faceted Browsing and Data Portraits". Damn! They are really doing nice work. I really encouraged him to evaluate their prototype because I think that it has a lot of possibilities. They are concentrating a lot of information and I think that they need to get some feedback now. Also, it was really nice to hear that he got a lot of inspiration from the paper "Attention please! Learning analytics for visualization and recommendation" by Erik Duval. You feel somehow lucky and think: "Yeah! And I am doing the PhD with him". Afterwards, you wonder why he hasn't still fired you after questioning all his ideas in the PhD meetings... I guess that it's the PhD student syndrome, we think that we know more than we actually know... or maybe it's my personal syndrome but I feel better thinking that others share the same problem... as the saying goes:

"It is a fool's consolation the think everyone is in the same boat"

Cheers! ;)

Preparing LAK'12 presentation!

I was thinking to write some lines about my presentation in LAK'12 conference. This presentation is about the paper Goal-oriented visualizations of activity tracking: a case study with engineering students and just now taking a look to the program, I saw that it will be broadcasted by video streaming. It is getting funnier, before I was just worried about giving a presentation with approximately one hundred attendees... now it will be broadcasted... so... more people... even more, I guess that it will be recorded so... more fun? :-) anyway... summarizing... it is a bit scary, isn't it? I think that I'll never get used to present my work, although it's always a nice learning experience (once finished :-))

However thinking about it, my conclusion is that I don't have anything to worry about... because I have a super presentation thanks to the feedback got from my colleagues in one internal try-out presentation.

I would like to share it with you and get maybe some additional feedback? I am still not convinced about mixing bar charts and box plots to show similar information, however the SUS questionnaire is not represented using box plots because the result was a bit weird. Anyway any feedback is welcome.

So... here you go!

View more presentations from José Luis Santos.

Twitter, blogs, tinyarm and a possible experiment

I have been playing with graphs libraries to visualize social relationships... you know this kind of stuff that nobody has done so far... (irony? ;)) So yes... I was playing with these libraries and I was thinking in which kind of experiment could involve these visualizations.

I haven't really thought about it in depth, but I reminded a conversation with Luis de la Fuente (when he was visiting us) that they (he and Katrien) were thinking to set up an ethnographic experiment with twitter and do some kind of comparison between belgian and spanish behavior... after some time  brainstorming we didn't see the point... and we finished with the unusual (irony again) sentence: "we have to think more about it" and we didn't talk about the topic anymore...

But now just thinking, an idea came up to my mind... I was thinking that we could set up an experiment four different research groups in different countries(?). The scope could be master thesis supervision.

What would be the requirements?

• Similar approach using social networks in the supervision of master thesis. For instance, twitter, blogs and Tinyarm to report the read/skimmed papers (It is not an hypothetical example... is how we supervise ours ;))
• The topic of the research groups should be similar.
• The location of the groups could be two from the center of Europe (Belgium-Germany) and two from the north of Africa... sorry... south of Europe (Portugal-Italy-Greece-Spain) (The order of the countries was not random). But preferably countries that share the root of their languages, so Greece could be discarded.
• All the communication should be in English.
• All the groups will share the same hashtag in Twitter. 
• All the papers in TinyARM should be also in English.

What could we study?

• Are new established links between students in different countries?
• If yes, makes sense to think that there are some intercultural relation?
• Maybe is more important the topic.
• In which system do the students create more new relationships? For instance, in twitter sending tweets to each other, commenting on each other blogs or even in TinyArm reading papers that others have read or recommending papers. 

And for sure... STEP UP! could be the central point of information... ;)

I guess that it is a very open experiment where we can study many factors, for sure, we have to think more about it. Damn! The forbidden sentence... now such experiment is condemned to obscurity... Anyway... it was a nice exercise to write it down...

Keep in mind that you are the only one who can avoid the predictable and sad fate of this experiment! ;)

Feedback is welcome, for sure, I can learn from you! :) Thx in advance! 

Meeting with one of my assesors

Wow! I was a bit... I don't know the correct word... afraid? I arranged a meeting with Andrew Vande Moore to sign my PhD Plan. Would he criticize too much my PhD Plan? Ok... I know... the PhD Plan went through several iterations with Erik and Katrien, but still I was aware that the PhD was a bit open to face my next two years of PhD.

I am working in learning... and learning is affected by multiple factors. We can control some of them but maybe others we are not aware of them (or we can not control them because are externals). But sometimes I am a bit afraid... I have a really good scenario to get a good PhD. We are experimenting with real students, so I could do really nice experiments, however, I end up after every evaluations with a feeling that I could have got more valuable evaluation (It also relates to my non-sexual usual sadomasochist tendencies, I don't know why but I'm always thinking that I did something wrong). 

Anyway, let's go to the conversation. In the last post I wrote about concepts such as awareness, meaningfulness and usefulness. So after the conversation, I added two concepts to my TODO list: Trust and Robustness. Btw, sometimes, I have the feeling that my thesis will be like the collin dictionary, a collection of concepts with their context-dependent definitions. Or maybe it's end up as an onthology... a self-definition of concepts that nobody uses except the owners (Oooops! Sorry! I don't want to offend pro-onthologists ;))

Andrew pointed out these concepts as something that I should consider in the evaluations. Why? Because it is a really important part in the learning process. An iterative feedback cycle between students and teachers. If students don't trust the teacher, would teaching make sense? We are trying to increase the awareness of our students through some kind of feedback (STEP UP!-the dashboard for those that don't remember the name ;)). I really like this picture, trust is reliability plus delight, and in this case, reliability relies (I know it's redundant) on robustness. How will students increase their awareness if they don't trust the system? And just thinking, the thesis students scenario is ideal for evaluating this. We have students, we have supervisors, and we have the activity of both in the social networks. In addition, neither students nor supervisor have the same motivation to work on the topics. Still master students don't have seen the visualizations so we have the perception of the students on how they are performing before the dashboard. We can show them the dashboard and we can ask them about their perception after the reflection. Afterwards, we can ask supervisors about their perception on how the student is going, and,  finally, we can mix motivation on the topic and social-network activity. Here are included posts, tweets and read/skimmed papers but even more important the comments received on their blogs from their peers and supervisors. Therefore these parameters can show us whether they are performance indicators. Because it was other part of the conversation and it also relates to trust:

What do we visualize?

It also relates to visual storytelling concepts. What is the message? What is the goal of the visualization? I think that I already mentioned before in other posts this concept. Andrew has explained to me how they use visualizations to display spent energy. He has told me that makes no sense to show spent energy whether they don't explain information about the context, for instance, how the sensors are and some additional contextual information.

We talked about more things but I was trying to summarize, although as you can see... I'm not good on it!. Also he offered me an additional testbed, he recommended to read one paper.

So conclusions of the meeting: It was a really productive meeting!

Usefulness, meaningfulness and awareness

Previous week, I had a couple of discussions with my colleagues Sten and Gonzalo about these terms. One common topic in our research is awareness... we try to increase the awareness of the user about what s/he is doing, and we use the (sub-)community to contextualize such activity. To this end, Gonzalo uses activity streams in Tinyarm (Haven't you tried it yet? Do it! It's a really cool tool for research awareness) and I visualize activity streams (Post, comments, twitter, toggl and soon Tinyarm activity).

Ok, both different methods to achieve awareness... but how to evaluate whether our tools provide awareness or not? Most of the papers, that I've read so far, made evaluations about conclusions extracted from the visualization or the tool in general... from my point of view, it is related to meaningfulness of something. Sure! It is a really important step! The user reflect on something meaningful and get aware(?) about their conclusions...

(?) Can we say that someone gets aware of something whether there is no change in her/his behavior?

(?) What is the proof that someone gets aware?

From my point of view, change of behavior can be a proof of awareness, however, someone can change the behavior for different reasons. No change in the behavior does not mean anything... maybe s/he feels ok with her/his behavior and s/he does not have such change... or maybe yes, but the trigger is not enough strong.

So what is a solution?

Sten sent me several user experience/design models (1, 2 and 3) previous week and my conclusion is that we should take a more pragmatic point of view. All of them share (directly or indirectly) concepts such as requirements, satisfaction and user's needs. Concepts that are also related to marketing. And ok... we cannot claim that we satisfy a predefined user need if the user comes back to our tool and use it... but at least, we know that something is going well when the user does it. We cannot claim that all the users share the same needs and we cover them... but we can say that we are providing a service and the users consider it useful for some reasons.

I have to think further on this and even more important, thinking about metrics that can be meaningful for us... I believe that usefulness (not perceived usefulness) is the key.

If anybody have some useful pointers, don't hesitate to tell me about them!

Step Up! Because sometimes the name really matters....

Sometimes I feel a bit... how can I say it? Silly... We try to find a suitable name for our application... Yes! Because marketing is also important in research... in fact, I think that is becoming more and more important... sometimes I think that is even more important than the research itself... but anyway... I drop this ideas off and start to discuss about the name itself.

 So... yes... sometimes I feel a bit weird because we discuss about names before we have our application in a stable version, even before we know that the concept works...

 Yes... it seems obvious sometimes that the concept works... For instance, in our case, most of the people thinks that increasing your awareness is positive! But how you will achieve this goal and whether the people will have enough time to spend on your proposed solution is not so clearly very often.

  Ok... so nobody understands why we look for a name in a so early stage... but it has an explanation... a name can define ideas... can send a message... can explain a concept... for me, a name is like an slogan in an election campaign... Would Obama have won without the slogan "Yes, we can!"? I don't know but it's clear that it had a clear impact on the people.

  So we have been thinking on different names such as:

1. LYA -> Learning from your activity.
2. Step Up!
3. SuP! -> Step Up!
4. Learnograph

  and other different names that can be representative for our application. But the selected one is:

 And maybe the question now is: What does it means for me?

STEP UP YOUR AWARENESS!

Because we are too worried about achieving goals, getting certificates and so on. Out of the educational context is the same... what really matters is to buy a new car, new house, new phone... but what happens with the process? Maybe if we were more aware about the process... our choices, goals and achievements would change...

And this is what STEP UP! tries to change telling to the students:

STOP! LOOK WHAT THE OTHERS ARE DOING!

ARE YOU IN THE CORRECT PATH?

THINK ABOUT IT!
MAYBE YOU CAN LEARN SOMETHING
MAYBE YOU CAN APPLY WHAT YOU LEARNT

And it is all what means STEP UP! for me. It is a tool for students, a tool for teachers, a tool for users that want to enjoy the process instead to achieve something, because in such process is where we spend more time... The success, the achievement, the goals are ephemeral... but we always keep our learned lessons along our life...

Blogs viz and twitter aggregation

After a while, I have just decided to post about the new visualization of blog posts/comments and the twitter integration.

Usually the problem is that depending on which decisions you took (related to architecture/development), they have some consequences on the next iterations, so you have to stop and to think a bit about it.

Now, we are working on visualizing blogs of our thesis students, but they have also to tweet. So the ideal situation is merge both visualizations trying to get an overview of the students activity. You can see the visualization here. And you can find a small screencast bellow.


You have two tables. The first one for the authors and the other for external people, they can be supervisors, colleagues of the authors or simply people that decided to comment on the blogs. You have a legend with three colors (Green means posts, Blue means comments and red means no activity at all).

You can interact with the table in different ways:

• If you click on the headers, the rows are sorted based on the activity.

• Clicking on the header with the right button of the mouse, you can access to the blogs.

• If you click on the cells, you get a tip with extra information. Click again to hide it.

• If you click on the sparklines, you get a bigger visualization that shows the aggregated data of comments and blogs per week of the year.

Now the question is:

How to add twitter information to the current table?

Just thinking that in the end, twitter in this visualization can be like a blog. An additional source where users can do some activity.

However, also we are thinking to aggregate other data from toggl. Ok... it can be another column. And every additional source can be an additional column, but in the end, you can find a huge overview of your classroom activity.

For instance, another use case that I'm thinking about is on one course that I'm teaching. Is about learning management systems and technological resources. The students learn how to manage different LMSs in several weeks. In this period, they have to tweet and blog about web 2.0 tools and/or reflect on their weekly activity. So other scenario could be something like blog activity, twitter activity and activity in the different LMSs. Something like number of activities, resources, lessons and so on. But it is always the same... not sure whether the column system is a good approach. In addition, they are more than 100 hundred students... it means 100 hundred rows in the table... ok... we can minimize the impact adding a filter and sorting functionality, but sometimes you lose a bit of the overview. Maybe the solution is to add additional metrics to take the overview, but the matter here is... what are meaningful metrics? Once that you decide to provide metrics, you are trying to drive the conclusions. As a teacher, maybe this is the goal. But not sure whether students expect that from a learning dashboard and what is the data that they want to look into. Yes... for sure... you can ask them, but it is not so obvious approach because sometimes they never thought about what is meaningful for them. Anyway, life is a matter of decisions... and wrong decisions means a lesson that can be learned for the future... as Erik usually says life is learning.

I will tell you more about my decission and what I have learnt in the next post! Because it is what really matters... What do we learn from our research...

After two years... preparing Phd Plan! (4)

This is my last post regarding the PhD Plan... after this post, I will try to formulate it in a formal way... but before I will answer two questions:

What I am doing now?

Now I am working in more widgets. Our professor, Erik, works with social networks in all his courses, specifically he suggests to the students that they have to tweet and to blog once per week. For teachers and assistants is relatively easy to follow what the students are doing whether they are actively participating in the social networks, for instance, using Google reader for the blogs and your twitter client, but becomes more difficult whether the teacher/assistant tries to know who is actively participating... and visualizations can play an important role to help this situation...

We developed this widget that visualizes per week the number of tweets:


On the top of the table, you can see the number of the week, on the right side the usernames. You can take an easy overview.

However, doing the same for blogs becomes a bit more close because you have to visualize posts but also comments who is posting and where.

These visualizations are not only oriented to teachers, also for students can be useful to be aware what others are doing and whether you are not enough active compare it with other students. One of the main critiques could be... to game the system is easy... you do a silly comment in the specific network and the cell becomes green. This is true... but we should not forget that teachers and assitants have to read all this social network interaction and they can remind students the kind of expected contributions.

So here maybe, we have to evaluate whether this kind of visualizations shared by teacher and students are good triggers for students to reflect on their progress.

Summarizing:

- How can we help students and teacher to drive conclusions using visualizations?
- How can we motivate students to reflect on their activity?
- What visualizations are more useful in a learning context and why?

So let's start to work!

After two years... preparing Phd Plan! (3)

Yes! Paper accepted! So someone of us (Sten Govaerts, Katrien Verbert, Erik or me ) will go next year to LAK Conference. They are really good news, it is a conference indexed in acm.org.

But the question is:

- What did we do to be accepted in this great conference?

It is an easy answer! We tried to explain how we proceed with our first attempt to answer one of the research questions:

- How can we help users to drive conclusions in a generic way?

We thought that term of learning goals could be nice. But what does learning goal means? It is a more difficult question... so we decided to reduce a bit the abstraction level of the word. Let's call it "Goal". What can be a goal? Everything that means achievement... a milestone in the course can be a goal? Sure! And if a students want to spent more or less time focused on an application, is it a goal? Yes! Sure! What am I trying to explain here? That we keep this word a bit away of the pedagogical formal meaning.

We got some inspiration from tools such as Rescuetime, Wakoopa, RunKeeper, JoggyCoach,... applications more oriented to the behavior rather than an abstracted goal. And why? For instance, in RecueTime, you can set up goals such as I want to be more productive! And you start to track your activity... but how RescueTime calculates your activity? They have a classification that scores every application as a productive or non productive... so browsing is not a productive application. It means that you can be editing a deliverable in google docs and you are not productive... sorry dude! You are wasting your time doing such deliverable... ;) Luckily, this classification is customizable... but anyway... blogging can be a non-productive tool and productive depending of the purpose... And It is really difficult to track... but also difficult to get input every time from the user that changes the focus to another application. So let's decrease the level of complexity and we ask:

Do the goals of the course help to contextualize the visualizations?

We developed this dashboard:



We developed a dashboard with visualizations of activity data. The overall goal of this dashboard is to enable students to reflect on their own activity and compare it with their peers. The time spent with different tools, websites and Eclipse IDE documents are tracked by RescueTime and the Rabbit Eclipse plug-in. The collected information is displayed in a dashboard containing goal-oriented visualizations. In the visualizations, the students can filter by different criteria, such as course goals and dates. Such filters allow contextualization of the visualized data for the user. Linking the visualizations with the learning goals can help students and teachers to assess whether the goal has been achieved.

In this course, the students have to develop software and go through the different phases of software development process, such as design, programming and reporting. To this end, they use tools such as LibreOffice, the Eclipse IDE and Mozilla Firefox. They have to share tasks and responsibilities between group members. Controlling the risks and evolution of such tasks is part of the assignment.

Visualization 1 are the goals during the course that can also be defined as milestones. Green color is that the time has expired and blue is that currently is going on.

Visualization 2 is a motion chart where x is the activity of the user and axis y is the average of the members of the team.

Visualization 3 is a timeline that visualizes the time spent on the different activities. To classify this activity we use the Rescuetime taxonomy that the site provides.

Visualization 4 is a barchart that visualizes the same than the previous one but visulized with the total activity.

Visualization 5 is a timeline that visualizes the time spent on the concrete tool.

Visualization 6 is a barchart that compares your activity with the total activity of the group.

Visualization 7 is a barchart that show the time spent on the different document on your Eclipse.

Finally, visualization 8 visualizes the time spent on the different websites.

At the beginning of this month we attended the Quantified Self conference in Amsterdam and there we learn something that was relatively useful... an easy way to present our results answering three simple questions:

- What did you do?
- How did you do it?
- What did you learn?

We have explained more or less the first two questions but...

What did we learn?

The perceived usefulness by the users is good but they don't use the application. It points out that we are doing something wrong... maybe the problem is that we don't stimulate enough the students... there are a behavior model that describes when the people modify their behavior. People needs triggers that motivate the change and the ability of the people to do the action. In this website, Fogg describes Facebook as a good trigger generator... for instance, when someone is tagged in a photo and receives the notification, automatically s/he connects to facebook to see the photo. Maybe this is some of the topics that we should explore...

How can we motivate the students to use learning analytic tools?

And tomorrow a bit more! What we are currently working and future plans!