Monday, 20 February 2012

Usefulness, meaningfulness and awareness

Previous week, I had a couple of discussions with my colleagues Sten and Gonzalo about these terms. One common topic in our research is awareness... we try to increase the awareness of the user about what s/he is doing, and we use the (sub-)community to contextualize such activity. To this end, Gonzalo uses activity streams in Tinyarm (Haven't you tried it yet? Do it! It's a really cool tool for research awareness) and I visualize activity streams (Post, comments, twitter, toggl and soon Tinyarm activity).

Ok, both different methods to achieve awareness... but how to evaluate whether our tools provide awareness or not? Most of the papers, that I've read so far, made evaluations about conclusions extracted from the visualization or the tool in general... from my point of view, it is related to meaningfulness of something. Sure! It is a really important step! The user reflect on something meaningful and get aware(?) about their conclusions...

(?) Can we say that someone gets aware of something whether there is no change in her/his behavior?

(?) What is the proof that someone gets aware?

From my point of view, change of behavior can be a proof of awareness, however, someone can change the behavior for different reasons. No change in the behavior does not mean anything... maybe s/he feels ok with her/his behavior and s/he does not have such change... or maybe yes, but the trigger is not enough strong.

So what is a solution?

Sten sent me several user experience/design models (1, 2 and 3) previous week and my conclusion is that we should take a more pragmatic point of view. All of them share (directly or indirectly) concepts such as requirements, satisfaction and user's needs. Concepts that are also related to marketing. And ok... we cannot claim that we satisfy a predefined user need if the user comes back to our tool and use it... but at least, we know that something is going well when the user does it. We cannot claim that all the users share the same needs and we cover them... but we can say that we are providing a service and the users consider it useful for some reasons.

I have to think further on this and even more important, thinking about metrics that can be meaningful for us... I believe that usefulness (not perceived usefulness) is the key.

If anybody have some useful pointers, don't hesitate to tell me about them!

Thursday, 9 February 2012

Step Up! Because sometimes the name really matters....

Sometimes I feel a bit... how can I say it? Silly... We try to find a suitable name for our application... Yes! Because marketing is also important in research... in fact, I think that is becoming more and more important... sometimes I think that is even more important than the research itself... but anyway... I drop this ideas off and start to discuss about the name itself.

 So... yes... sometimes I feel a bit weird because we discuss about names before we have our application in a stable version, even before we know that the concept works...

 Yes... it seems obvious sometimes that the concept works... For instance, in our case, most of the people thinks that increasing your awareness is positive! But how you will achieve this goal and whether the people will have enough time to spend on your proposed solution is not so clearly very often.

  Ok... so nobody understands why we look for a name in a so early stage... but it has an explanation... a name can define ideas... can send a message... can explain a concept... for me, a name is like an slogan in an election campaign... Would Obama have won without the slogan "Yes, we can!"? I don't know but it's clear that it had a clear impact on the people.

  So we have been thinking on different names such as:
  1. LYA -> Learning from your activity.
  2. Step Up!
  3. SuP! -> Step Up!
  4. Learnograph
  and other different names that can be representative for our application. But the selected one is:


 And maybe the question now is: What does it means for me?

STEP UP YOUR AWARENESS!

Because we are too worried about achieving goals, getting certificates and so on. Out of the educational context is the same... what really matters is to buy a new car, new house, new phone... but what happens with the process? Maybe if we were more aware about the process... our choices, goals and achievements would change...

And this is what STEP UP! tries to change telling to the students:

STOP! LOOK WHAT THE OTHERS ARE DOING!
ARE YOU IN THE CORRECT PATH?
THINK ABOUT IT!
MAYBE YOU CAN LEARN SOMETHING
MAYBE YOU CAN APPLY WHAT YOU LEARNT

And it is all what means STEP UP! for me. It is a tool for students, a tool for teachers, a tool for users that want to enjoy the process instead to achieve something, because in such process is where we spend more time... The success, the achievement, the goals are ephemeral... but we always keep our learned lessons along our life...

Tuesday, 7 February 2012

Blogs viz and twitter aggregation

After a while, I have just decided to post about the new visualization of blog posts/comments and the twitter integration.

Usually the problem is that depending on which decisions you took (related to architecture/development), they have some consequences on the next iterations, so you have to stop and to think a bit about it.

Now, we are working on visualizing blogs of our thesis students, but they have also to tweet. So the ideal situation is merge both visualizations trying to get an overview of the students activity. You can see the visualization here. And you can find a small screencast bellow.


You have two tables. The first one for the authors and the other for external people, they can be supervisors, colleagues of the authors or simply people that decided to comment on the blogs. You have a legend with three colors (Green means posts, Blue means comments and red means no activity at all).

You can interact with the table in different ways:

  • If you click on the headers, the rows are sorted based on the activity.
  • Clicking on the header with the right button of the mouse, you can access to the blogs.
  • If you click on the cells, you get a tip with extra information. Click again to hide it.
  • If you click on the sparklines, you get a bigger visualization that shows the aggregated data of comments and blogs per week of the year.
Now the question is:

How to add twitter information to the current table?

Just thinking that in the end, twitter in this visualization can be like a blog. An additional source where users can do some activity.

However, also we are thinking to aggregate other data from toggl. Ok... it can be another column. And every additional source can be an additional column, but in the end, you can find a huge overview of your classroom activity.

For instance, another use case that I'm thinking about is on one course that I'm teaching. Is about learning management systems and technological resources. The students learn how to manage different LMSs in several weeks. In this period, they have to tweet and blog about web 2.0 tools and/or reflect on their weekly activity. So other scenario could be something like blog activity, twitter activity and activity in the different LMSs. Something like number of activities, resources, lessons and so on. But it is always the same... not sure whether the column system is a good approach. In addition, they are more than 100 hundred students... it means 100 hundred rows in the table... ok... we can minimize the impact adding a filter and sorting functionality, but sometimes you lose a bit of the overview. Maybe the solution is to add additional metrics to take the overview, but the matter here is... what are meaningful metrics? Once that you decide to provide metrics, you are trying to drive the conclusions. As a teacher, maybe this is the goal. But not sure whether students expect that from a learning dashboard and what is the data that they want to look into. Yes... for sure... you can ask them, but it is not so obvious approach because sometimes they never thought about what is meaningful for them. Anyway, life is a matter of decisions... and wrong decisions means a lesson that can be learned for the future... as Erik usually says life is learning.

I will tell you more about my decission and what I have learnt in the next post! Because it is what really matters... What do we learn from our research...

Tuesday, 20 December 2011

After two years... preparing Phd Plan! (4)

This is my last post regarding the PhD Plan... after this post, I will try to formulate it in a formal way... but before I will answer two questions:

What I am doing now?

Now I am working in more widgets. Our professor, Erik, works with social networks in all his courses, specifically he suggests to the students that they have to tweet and to blog once per week. For teachers and assistants is relatively easy to follow what the students are doing whether they are actively participating in the social networks, for instance, using Google reader for the blogs and your twitter client, but becomes more difficult whether the teacher/assistant tries to know who is actively participating... and visualizations can play an important role to help this situation...

We developed this widget that visualizes per week the number of tweets:


On the top of the table, you can see the number of the week, on the right side the usernames. You can take an easy overview.

However, doing the same for blogs becomes a bit more close because you have to visualize posts but also comments who is posting and where.

These visualizations are not only oriented to teachers, also for students can be useful to be aware what others are doing and whether you are not enough active compare it with other students. One of the main critiques could be... to game the system is easy... you do a silly comment in the specific network and the cell becomes green. This is true... but we should not forget that teachers and assitants have to read all this social network interaction and they can remind students the kind of expected contributions.

So here maybe, we have to evaluate whether this kind of visualizations shared by teacher and students are good triggers for students to reflect on their progress.

Summarizing:

- How can we help students and teacher to drive conclusions using visualizations?
- How can we motivate students to reflect on their activity?
- What visualizations are more useful in a learning context and why?

So let's start to work!

Monday, 19 December 2011

After two years... preparing Phd Plan! (3)

Yes! Paper accepted! So someone of us (Sten Govaerts, Katrien Verbert, Erik or me ) will go next year to LAK Conference. They are really good news, it is a conference indexed in acm.org.

But the question is:

- What did we do to be accepted in this great conference?

It is an easy answer! We tried to explain how we proceed with our first attempt to answer one of the research questions:

- How can we help users to drive conclusions in a generic way?

We thought that term of learning goals could be nice. But what does learning goal means? It is a more difficult question... so we decided to reduce a bit the abstraction level of the word. Let's call it "Goal". What can be a goal? Everything that means achievement... a milestone in the course can be a goal? Sure! And if a students want to spent more or less time focused on an application, is it a goal? Yes! Sure! What am I trying to explain here? That we keep this word a bit away of the pedagogical formal meaning.

We got some inspiration from tools such as Rescuetime, Wakoopa, RunKeeper, JoggyCoach,... applications more oriented to the behavior rather than an abstracted goal. And why? For instance, in RecueTime, you can set up goals such as I want to be more productive! And you start to track your activity... but how RescueTime calculates your activity? They have a classification that scores every application as a productive or non productive... so browsing is not a productive application. It means that you can be editing a deliverable in google docs and you are not productive... sorry dude! You are wasting your time doing such deliverable... ;) Luckily, this classification is customizable... but anyway... blogging can be a non-productive tool and productive depending of the purpose... And It is really difficult to track... but also difficult to get input every time from the user that changes the focus to another application. So let's decrease the level of complexity and we ask:

Do the goals of the course help to contextualize the visualizations?

We developed this dashboard:



We developed a dashboard with visualizations of activity data. The overall goal of this dashboard is to enable students to reflect on their own activity and compare it with their peers. The time spent with different tools, websites and Eclipse IDE documents are tracked by RescueTime and the Rabbit Eclipse plug-in. The collected information is displayed in a dashboard containing goal-oriented visualizations. In the visualizations, the students can filter by different criteria, such as course goals and dates. Such filters allow contextualization of the visualized data for the user. Linking the visualizations with the learning goals can help students and teachers to assess whether the goal has been achieved.

In this course, the students have to develop software and go through the different phases of software development process, such as design, programming and reporting. To this end, they use tools such as LibreOffice, the Eclipse IDE and Mozilla Firefox. They have to share tasks and responsibilities between group members. Controlling the risks and evolution of such tasks is part of the assignment.

Visualization 1 are the goals during the course that can also be defined as milestones. Green color is that the time has expired and blue is that currently is going on.

Visualization 2 is a motion chart where x is the activity of the user and axis y is the average of the members of the team.

Visualization 3 is a timeline that visualizes the time spent on the different activities. To classify this activity we use the Rescuetime taxonomy that the site provides.

Visualization 4 is a barchart that visualizes the same than the previous one but visulized with the total activity.

Visualization 5 is a timeline that visualizes the time spent on the concrete tool.

Visualization 6 is a barchart that compares your activity with the total activity of the group.

Visualization 7 is a barchart that show the time spent on the different document on your Eclipse.

Finally, visualization 8 visualizes the time spent on the different websites.

At the beginning of this month we attended the Quantified Self conference in Amsterdam and there we learn something that was relatively useful... an easy way to present our results answering three simple questions:

- What did you do?
- How did you do it?
- What did you learn?

We have explained more or less the first two questions but...

What did we learn?

The perceived usefulness by the users is good but they don't use the application. It points out that we are doing something wrong... maybe the problem is that we don't stimulate enough the students... there are a behavior model that describes when the people modify their behavior. People needs triggers that motivate the change and the ability of the people to do the action. In this website, Fogg describes Facebook as a good trigger generator... for instance, when someone is tagged in a photo and receives the notification, automatically s/he connects to facebook to see the photo. Maybe this is some of the topics that we should explore...

How can we motivate the students to use learning analytic tools?

And tomorrow a bit more! What we are currently working and future plans!

Monday, 12 December 2011

After two years... preparing Phd Plan! (2)

As I mentioned in my previous post, I shifted a bit my research after the ARIADNE Registry work...

I got involved in ROLE project. It is a project that research about Responsive Open Learning Environments. Specifically, it focuses on Personal Learning Environments (PLE). PLEs enable users to personalize the services required for the different assignments in a learning process, in a technical way, we can define it such as a mashup of services.

Don't hesitate to read our paper about ROLE that we presented at ECTEL'11!!!!!!!!!. The main authors of this paper were my colleagues Sten Govaerts and Katrien Verbert.

Personalization is the key. If users can personalize their own learning environment, we add another possible factor that can influence on the success or fail of the user activities... which resources and services can be the key to understand learners (from a teacher perspective but also from their own perspective).

The understanding of the learner is the key of my current work, the development of tools to reflect on the user activity and visualizations can be one of the possibilities.

The first attempt to do something in this field was this application and we submitted a paper to EFEPLE workshop. You can see a screenshot of the application below.



During the development, we did some evaluations with teacher assitants, and finally had a discussion in the context of the EFEPLE'11 workshop.

The results pointed out some interesting details:

1. Users have different visualization background and they use the visualizations for different purpose. For instance, some of the evaluated users highlighted the fact that bar charts are really easy to understand and they really like it. However, others user with higher skills in visualization tools asked for more relevant information. They were interested for more detailed information.

2. The viewer needs information about the context to understand the visualizations. If you know the purpose/goal of the learners, the activity, the course, the specific student... the visualization makes more sense... now it is a dashboard that visualizes information... but the question that came up is:

How can we provide more meaningful visualizations in a learning environment?

My colleague Joris Klerkx recommended me a paper about narrative visualzation that I really liked it. This paper explains in a few words that visualization can be a good support to explain a story and shows some examples.

Sometimes I heard the discussion about what is better visualizations or recommendations? Simplifying the differences (both are huge fields), one rely on the user the cognitive efforts to achieve conclusions (Visualizations) and the other minimize the user cognitive efforts providing some possible conclusions (Recommendations). Maybe it is not a matter on which approach is the best... depending of the user and depending the quantity and quality of data that we have, one can be better than the other...

So that paper showed me a middle term approach, in a narrative way, the writer was driving the conclusions of the reader and the visualizations were the support of our conclusions (It sounds really similar to me when we use visualizations writing a paper to support our conclusions with quantity data, isn't it?).

Again, my question that was coming up to my mind was:

How can we help users to drive conclusions in a generic way?


Generic way is the key of the question. The systems that I'm working with are very different, different services, different trackers, different quality of data, different formats, so... it makes a bit more complex driving conclusions if you don't know which data and context you will have... again... context is the key... and sometimes we don't have access to this context... for instance, PLEs, LMSs, learning environments in general don't enable external access to such information easily...

How can we provide information about the context?


I finish here my post... but don't worry... I haven't still solved these questions... maybe someday we can read my thesis with useful conclusions about this point... :-) but I think that this question is really interesting to research on: How can we help users to understand visualizations?

Another post comes soon... waiting to know whether my last paper is accepted or not... crossed fingers...

Friday, 9 December 2011

After two years... preparing Phd Plan! (1)

Yes... I know that it can sound a bit weird... It's something that I should have done during the first year of my PhD... but there is always a positive side... now I have deeper knowledge in my field. So now, I have to do a bit of inverse engineering for what I have got so far and a bit of planification for my next challenges... The goal of this reflection is finding the research question that I have answered and the research questions that I am going to answer in the near future.

So let's go to think about what I have so far...

At lirias you can find my publications at KULeuven.

In 2010, I got two publications. The first one was presented at ECTEL conference. It describes the ARIADNE architecture. My contribution to this paper is about the ARIADNE registry and it has a strong link with the second paper due to it is the description of the already mentioned ARIADNE registry that we presented at the SE@M workshop.

What is this registry about?

The ARIADNE Registry is an open source project that enables users and services to discover new repositories of learning resources. Yes... I know... it sounds interesting... isn't it? ;)

What can we do once that we have found a new repository?

First, I should point out that these repositories can work in a federated way... why? Because they can share descriptive information about the learning resources and, to this end, they expose this information through service interfaces. This idea is similar to the WEB 2.0 concept, different services that can interact with others, for instance, facebook has an application that if you install it, all your tweets are posted directly in your facebook wall. It's cool... updating one social network status, it appears in two social networks... so one time work... double impact of your status...

Let's apply the same concept to the repositories... you (as learning resource provider, simple user or teacher) update your learning object once and such information is extended along the network of repositories hence your information is accessible from different access points. (It's important that user remains the control of the object itself... so this federation is not incompatible with business models). The idea follows the same concept: increase the impact of your work!

It seems ideal the situation but...

What is the challenge?

The challenge is that repositories expose their information using different protocols and specifications. For instance, repositories can expose their information using SQI, SRU/W or OAI-PMH protocols and the description or metadata using specifications such as IEEE LOM, Dublin Core and RSS. Therefore, these requirements should be described somewhere enabling access to external services to this information automatically. In this way. they can understand how they can access to such metadata.

Looking for a possible solution in the context of ASPECT project, we used the specification IMS LODE. Such specification allowed us to define different collection of objects and define the protocols to access the repository information.

It worked quite well providing access to more than 700.000 metadata objects in total and the integration fo the ARIADNE Harvester with the ARIADNE Registry.

What are the limitations of this approach?

Currently, learning resources are not static objects such as docs and pdfs. Blogs, tweets, forums and so on can be learning resources. IMS LODE is an specification oriented to define protocols and specifications, however these WEB 2.0 services do not follow specifications and protocols, so most of the definitions are specific to such services and we can lose the process automatization.

What is the research question?

What are the benefits on defining services to share learning resources using IMS LODE?

As you will see in my next posts, I changed a bit my orientation on my research focused on how we can track users and enabling reflection on the activity for teachers and students, but following the same research line 'mashup of services'.