\section{Introduction}
Which factors contribute to people's \emph{perception of creativity} in artificial systems has been recognised as a central question in computational creativity (CC) research \citep{colton2008creative_tripod}. Insights into this question can not only support the fairer comparison of existing systems and the design of new systems, but also inform strategies to foster people's acceptance of AI systems as creative contributors.

By \enquote{the human perception of creativity}, we denote a person's subjective assessment of an AI system's creativity based on available evidence. Some of the factors identified or hypothesised to affect this perception -- assuming that the creative process and product remain constant -- are the human assessor's competence \citep{lamb2015} and technological optimism \citep{mumford15}, the system's motivational mechanism \citep{guckelsberger2017addressing} and the framing of its creative process \citep{charnley2012notion}, including the description of \enquote{life experiences}~\citep{Colton2020}. In this paper, we focus on a factor that has only recently been hypothesised to affect the human perception of creativity \citep{survey2021embodiment
: the system's \emph{embodiment} as -- roughly speaking -- the shape of its body and its means to interact with the environment \citep[cf.][]{ziemke2003s}.

A recent survey of publications~\citep{survey2021embodiment} at ICCC as prime venue of CC research attests a long-lasting, steady and recently increasing interest in engineering autonomous as well as co-creative, embodied CC systems in various creative domains. 
Crucially though, the survey has not found a single study offering generalising, empirical insights about the effect of a system's embodiment on the perception of creativity. This is highly problematic in that the effect could be substantial; some even hypothesise that, given a sufficiently large \enquote{embodiment gap} to the human observer, a system's creativity might remain entirely unnoticed \citep{guckelsberger2017addressing}. 

We pave the way towards gaining such insights by contributing a framework for conducting empirical studies on how a system's embodiment impacts the human perception of its creativity. The framework is intended for and relies on comparing this facet of human perception between two or more systems. Within it, we highlight the importance, and discuss the potential values and relationship of two independent variables: the \emph{participant's perceptual evidence}, and the \emph{system's embodiment}. Manipulating these variables, we can study what kind of observations about an artificial system lead to the attribution of creativity, and how. We support our discussion with examples, but focus on the domain of \emph{visual art} to remain within the scope of a short paper. 

Crucially, we target the effect of embodiment on the \emph{perception} of creativity, rather than on creativity per se, which would affect the creative process and product \citep{jordanous2016four}. To this end, we promote that it is essential to encapsulate the system and its perceptible qualities from the participants while keeping the creative behaviour constant between the compared systems. Naturally, isolating these effects may offer us insight on how a system's embodiment can be changed for it to \textit{appear} more creative, even though the underlying creative processes remain the same.

We next introduce our experimental study framework,
justifying the controlled and independent variables, discussing their possible values and providing recommendations for the measurement of creativity as dependent variable. We finish with conclusions and future work.

\section{Experimental Framework}

Our experimental framework rests on three assumptions of causality:~(1) A participant's assessment of a system's creativity depends on what aspects of the system and its interaction they can perceive. We thus factorise what we earlier denoted the \enquote{perception of creativity} into the perceived evidence, and its assessment. This perception in turn depends (2)~on the system's embodiment, and (3) the creative process which it executes. We consequently model two independent, one dependent and one controlled variable. We manipulate the compared \emph{system's embodiment} (independent variable, IV) and which aspects of the systems' behaviour the participants can perceive, i.e. their \emph{perceptual evidence} (IV). We measure the participants' \emph{perceived creativity assessment} (dependent variable, DV), and hold the underlying \emph{creative process}, which is reflected in the perceptual evidence, constant between conditions (controlled variable, CV). 

This framework imposes some restrictions on the studied systems. Namely, the creative process of the system must be kept the same while altering the embodiment or the perceptual evidence. Moreover, altering the perceptual evidence 
may be impossible in domains where the embodiment itself is part of the observed artefact, such as in dance. 

We next provide instructions for instantiating and executing a specific experiment from this framework, followed by a discussion of each variable and its potential values. 









\subsection{Instantiating an Experiment}

To instantiate an experiment for a specific research question, one must make the following decisions:


\begin{itemize}
    \item Whether to manipulate only one or both independent variables (system embodiment, perceptual evidence)
    \item Which values to assume for these independent variables. To yield more than one condition for a comparative measurement, it is crucial to select at least two or more different values of perceptual evidence, or embodiments.
    \item Which creative process to perform as the controlled variable, held constant across the conditions.
    \item How to measure the assessment of creativity as dependent variable: questionnaire, interview, think-aloud; potentially complemented with post-study questionnaire or physiological measurements.
    \item Whether to expose each participant to one, all, or some conditions (between/within/mixed design).
    \item How to analyse the collected data (e.g. exploratory statistics and statistical significance testing for quantitative, thematic analysis for qualitative data).
\end{itemize}

By controlling the perceptual evidence, we can assess how strongly the subjective assessment of embodied CC depends on what data on the artefact-process-system whole has been gathered. This can also be interpreted as manipulating not the system's, but the participant's sensorimotor embodiment. The experiment also allows to study how embodiment affects the perceived creativity by altering embodiment (e.g.~virtual and physical) while keeping the process and the perceptual evidence the same. Both effects can be investigated together in a factorial design with appropriate analysis methods. Crucially, if the same participant is exposed to multiple conditions, the order effect must be mitigated by randomising the exposition order for each participant.


\iffalse
The general structure is as follows:
\begin{enumerate}
    \itemsep0em
    \item Select the creative process and its embodiment(s) as the subject of the study 
   
    \item Select the types of perceptual evidence (PE1-PE3) for the study (see below)
    \item Define what is meant by creativity and how it is assessed using questionnaires / interviews (see below)
    \item For each study participant:
    \begin{enumerate}
        \itemsep0em
        \item Select one or more embodiments of the creative process
        \item Select one or more PEs
        \item For each combination of embodiment and PE (in randomised order):
        \begin{enumerate}
            \item Let the participant observe the embodiment produce an artefact through the PE
            \item Let the participant answer questionnaire / interview about the combination
        \end{enumerate}
        \item Let the participant answer post-study questionnaire / interview which may ask comparison questions
    \end{enumerate}
    \item Collect all the answers and do appropriate analysis of the results depending on the quantitative or qualitative nature of the study
\end{enumerate} 

The general structure described above can be altered in multiple ways. The important part is to select at least two different type of perceptual evidence, or two different embodiments of the same creative behaviour, so that a comparative analysis between them is possible. 
\fi


\subsection{Experiment Procedure}
The experiment should be executed for individual, and not groups of participants, in a lab setting without distractions. After receiving an introduction to the experiment, the participant must provide consent on the use of their data and provide demographic information. They are then exposed to perceptual evidence of a single embodied system (i.e.~process and embodiment). Data is recorded during or after this exposure, depending on the selected measurement. The experiment can stop (between), commence with all remaining conditions (within) or a subset (mixed), depending on the choice of experimental design. It can be concluded with a post-study questionnaire and debriefing session.

\subsection{System Embodiment (IV)}
Our study allows for comparing people's perception of multiple systems, each with a different embodiment. Amongst others, this enables the comparison of very varied embodiments for stronger effects, some of which could not possibly be expressed on the same hardware. One independent variable defines the embodiments to be compared, with each value corresponding to one system and embodiment.

While \enquote{embodied} is often considered synonymous with \enquote{robotic}
systems, cognitive science has distinguished several more types of embodiment, which lend themselves naturally as values of our independent variable. We adopt the typology by \citet{ziemke2003s}, which has been slightly extended by \citet[][highlighted in \emph{italics}]{survey2021embodiment}:

\begin{description}
\item{\textbf{structural coupling}}, characterising systems that can perturbate, and, vice versa, be perturbated by their surrounding environment \citep{varela1991embodied}. 
\item{\textbf{historical}, characterising systems whose present state is the result of a history of such structural couplings, developed through interactions with the environment over time  \citep{varela1991embodied, ziemke1999rethinking}.}
\item{\textbf{virtual}}, \emph{characterising simulated systems embedded in and distinguished from a simulated environment. The virtual body can affect the environment and vice versa.
\item{\textbf{physical}}, characterising systems with a physical body \citep{brooks1990elephants, pfeifer2001understanding} that can interact with the environment by being subjected to and exercising physical force. This includes most robots.
\item{\textbf{organismoid}}, characterising \emph{virtually} or physically embodied systems with the same or similar form and sensorimotor equipment as living organisms. Approximations of humanoid embodiment can be considered a subset.
\item{\textbf{organismic}}, applying to living \emph{and artificial} systems capable of autopoiesis, i.e. of maintaining their internal organisation and surrounding boundary against internal and external perturbations by means of self-producing processes \citep{von1920theoretische, maturana1987tree}.
\end{description}
While structural coupling establishes the baseline for what can be considered an embodied system, historical embodiment practically characterises any non-theoretical system. Consequently, we suggest to only consider the latter four types and their sub-types for experimentation. A specific system can instantiate several types, which can be combined as values of the independent variable. For instance, one might be interested in comparing a system with virtual+humanoid (a subset of organismoid) with a system implementing physical+humanoid embodiment. 

While these types provide some initial orientation, they would typically be differentiated further. For instance, robots in the shape and with similar sensorimotor equipment to an ant, a dolphin, or a bird all share the same type of physical, organismoid but non-humanoid embodiment. While such diverse comparisons are possible, experimenters might also be interested in measuring the effect of subtle changes to the same embodiment, e.g.~equipping a humanoid, physically embodied system with a different hand.

The choice of values in a specific experiment will likely be informed by the overarching research question and the systems at hand. If no requirements are imposed, we argue that the most value to the community at present can be created by comparing those types of embodiment that most strongly differentiate existing CC studies: \emph{virtual} vs. \emph{physical} embodiment, followed by humanoid vs. non-humanoid, organismoid embodiment \citep{survey2021embodiment}.

\subsection{Perceptual Evidence (IV)}

We hold that how a person observes an embodied system's creative process, yielding different kinds of \textit{perceptual evidence} (PE), plays a key role in their attribution of creativity. This is why our framework allows to manipulate the \textit{perceptual evidence} (PE) as IV. We propose three values: the finished artefact (PE1), the artefact as it is being produced (PE2), and the embodied system as it is producing the artefact (PE3). In all cases, the participant is assumed to remain in a fixed location. Further values could comprise different observation angles, or allowing the participant to observe freely. We next discuss how each of the proposed values might affect the attribution of creativity as DV, using the four perspectives on creativity framework~\citep{jordanous2016four}. 

\paragraph{PE1: The artefact} The participant only observes the produced artefact, but not the creation process or the creating, embodied system. While the artefact evidences an embodied system's historical, sensorimotor interaction with its environment, the participant can only speculate on the details of its embodiment. This can serve as the baseline to which other types of perceptual evidence are compared.\\%From Ziemke's~(\citeyear{ziemke2003s}) embodiment typology, this would attend for evidence of historical embodiment, i.e. someone has made the artefact but the study subject can only speculate (based on the qualities observed from the artefact) about the process, the embodied system and their properties.
\textbf{Creativity:} Observing only the artefact allows assessing creativity from the \emph{Product} perspective. While not observing the production makes it challenging to assess creativity from the \emph{Process} perspective, some hints of the \emph{Process} or \emph{Producer} qualities (e.g.~skill, imagination) may be evidenced in the artefact, becoming available for observation.


\paragraph{PE2: The process} The participant observes the creation process of the artefact, e.g. lines drawn on the screen, and the finished artefact. The embodiment of the system however is not revealed. This is straight-forward when dealing with modifications of the same embodiment type, but comparisons between different types, e.g. virtual and physical embodiment, require more sophisticated setups. While the process is controlled and should thus not reveal a particular embodiment, we yet hold that observing it might bias the participant towards assuming a particular kind of embodiment. Investigating PE2 allows to account for such effects.\\
\textbf{Creativity:} Allowing the participant to observe the artefact as it is being produced enables (in suitable domains) assessing creativity from the \emph{Process} perspective. However, the full (creative) process of the system is not observable as the embodiment is left out. Using PE2 as an intermediate step between PE1 and PE3 allows comparing, e.g., how observing the artefact creation process itself affects the assessment of the produced artefacts (comparing PE1 and PE2).

\paragraph{PE3: The system} The participant observes the embodied system while it is producing the artefact. In contrast to PE2, the participant can observe how the embodied system actually operates, e.g.~drawing lines on a canvas. This allows for including all relevant embodiment types in the participant's observations, although organismic embodiment might be hard to realise within a sensible experimental time frame.\\
\textbf{Creativity:} Observing the full system allows the participant to assess its full (externally visible) \emph{Process}, and observing the embodiment may give hints about the system's properties from the \emph{Producer} perspective. Comparison (of the same system) between PE2 and PE3 allows studying the effect of observing the embodiment itself on the perceived creativity of the \emph{Products} and their production \emph{Process}.


\subsection{Perceived Creativity Assessment (DV)}
\label{ssec:assessing_perceived_creativity}

Depending on the goals of the experiment, a participant's perceived creativity assessment can be measured either qualitatively or quantitatively. While a comprehensive overview of creativity measurement practices is out of scope for this paper, we make the general recommendation to use standardised and CC-specific assessment practices such as SPECS~\citep{Jordanous2012SPECS}. Here, the study designer first specifies what is meant by creativity and then defines the criteria for how it can be (perceptually) assessed. Moreover, combining observations (by the experimenter) and self-reporting of the participants has been known to produce apt results in studies assessing human experience while using creative systems~\citep{kantosalo2019quantifying_co-creative}. 

If conducting an interview, the study designer must pay close attention to the formulation of their questions in order to foster insights on a particular aspect of the experiment, e.g. the observed artefact, its creation process, or the system. Moreover, any question must explicitly distinguish between the observed evidence and unobserved variables: a~participant exposed to PE1 can be asked about the embodied system's properties but it should be clearly communicated that answering the question requires them to imagine system properties which they have not directly observed.


\iffalse
\begin{itemize}
    \item Good rule of thumb is to use SPECS~\citep{Jordanous2012SPECS} \christian{But it's also utterly underspecified...} \anna{I wouldn't say underspecified, just that there is not much work on how to translate these definitions into questionnaires. Some talk about developing questionnaires for measuring creativity related experience is in my work in Connection Science or Digital Creativity}
    \item What kind of questions are asked from the participants? Do they deal only with individual systems or are the participants also asked to compare the systems they have been exposed to (in case a single participant is exposed to more than one level of perception).? \anna{Comparisons work great, but there may also be a novelty, or ordering effect. See my work, where children, when they got more familiar with the task, started enjoying themselves more, making the last tested writing method significantly more enjoyable than the first (Kantosalo, A., & Riihiaho, S. (2019). Quantifying co-creative writing experiences. Digital Creativity, 30(1), 23-38.) Comparisons are also known to make more sense when people are asked about emotional responses to anything, see (Yannakakis, G. N., Cowie, R., & Busso, C. (2017, October). The ordinal nature of emotions. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 248-255). IEEE.)}
    \item Can we also evaluate the participants response (e.g. by recording the participant while they are observing the system and then analysing the recording), not just the questionnaire/interview? \anna{Based on my experience, I advocate for combining observation and self-report. We also mention in the Digital Creativity paper that while some observational evidence is clear, sometimes the self report is different from this.}
    \item \simo{Anna perhaps has the best knowledge on these.} \anna{I can add a few ideas if we agree to keep this here. But I agree, maybe some more specificity is needed. If nothing more, then choose some samples from SPECS to show how to formulate questions.}
\end{itemize}
\fi

\subsection{Creative Process (CV)}
In order to measure the effect of embodiment on the perception of creativity, rather than creativity per se, we must keep the creative process constant between conditions. An \enquote{online} solution requires alterations to the system to produce equal action sequences between runs, e.g.~fixing random seeds, setting learning rates to zero, etc. This still allows the system to condition its actions on previous sensor states in a closed loop, potentially producing a process which incorporates its present environment. If such a solution is not feasible, an \enquote{offline} solution can be used instead. Here, all process-constituting actions can be pre-recorded from a normal system run and replayed for different conditions. Crucially though, running the system in such an open-loop fashion might affect the human perception of its creativity.

While keeping the internal action signals constant, keeping the outside process as perceived by the participant the same may not always be possible. In particular in physically embodied systems, actuator and sensor noise and low fidelity will yield slight variations in the observable process, and potential changes to the resulting artefact. How sensitive the measurement of the DV is to such fluctuations is an open research question in its own right. We recommend to account for it by repeated measurements of the same condition and ensuring that the variations only produce minuscule differences to the produced artefacts.

Lastly, we anticipate that the human assessment of creativity might be affected by many latent properties of the creative processes, e.g. its length, execution speed, or complexity. To account for such biases, we can evaluate multiple creative processes with varying properties, effectively turning the creative process into an independent variable.



\begin{comment}
\section{Discussion}
\christian{Moved the creative process as controlled variable up; do we want to keep this or remove it altogether? Two possibilities to discuss: (i) further extensions to framework; (ii) how to apply framework on existing system outside the lab, to increase the ecological validity of the findings. NOTE: if dropping (i), adjust the last sentence of the introduction! If removing the discussion altogether, remove "core" from previous section title!}

Different domains and artefacts observed by different modalities affect the experiment setup. We have focused on visual perception, but also other senses -- and types of evidence -- can be included. For example, how is robotic movement perceived if it is accompanied with music or silence? Moreover, we could control for the framing, i.e. a description of e.g.~the motivation and steps involved in the production of the artefact accompanied with information about the system that may not be directly observable.

\anna{There could be a note about different modalities here. We talk about modalities in (Kantosalo, A., Ravikumar, P. T., Grace, K., \& Takala, T. (2020, September). Modalities, Styles and Strategies: An Interaction Framework for Human-Computer Co-Creativity. In ICCC (pp. 57-64).) where we consider modality one of the key facets in defining and designing interaction with a co-creative algorithm. It could be extended here, with the core disambiguation of modality as sensory modality + information channel + input/output device}
\end{comment}


\begin{comment}


\subsection{Controlled Variables}
\simo{Title subject to change. The idea is to cover here briefly the most prominent alterations (e.g. each participant is show X levels of perception in a random order, the artefact generated is not always the same, etc.) to the general structure of the experiment and what are their implications.}

The general structure of the experiment allows variability. We cover briefly a few prominent factors that can be altered and their possible implications. 

\subsubsection{Selected system(s)} \todo{Put the assumptions about the system here? Assumptions: process should be (near?) deterministic for the study, i.e. the overall process can include randomisation but for the study the particular artefact generation process should be as close the same as possible for each level of perception. We could also have more than one system and their perceptions could be compared.} 

\subsubsection{Artefact generation processes} The study can include more than one process (done by the same system) that leads to a generation of an artefact. The qualities of the generation processes and the generated artefacts should, however, be taken into consideration when designing the experiment. For example, if one of the artefacts is clearly more complex than others, it can lead to biases in the results if not properly taken care of. (Naturally, one of the experiment goals may be to study how complexity of the produced artefacts affects the perceived creativity on the different levels of perception.)
\christian{We must be more clear about why we would do modifications here and how the challenges can be accounted for. E.g. we would include different processes in order to randomize out any bias on the perception of embodiment caused by a specific process and the resulting artefact. We can mitigate differences e.g. in complexity by balancing these processes across all conditions / levels of perception, e.g. realising all processes on all systems.}
\anna{This starts to go towards other perception evaluations besides embodiment. The relationship between creativity and perceived/assumed evidence for it becomes stronger. We should be careful about this and think if we need to, or for the purposes of this workshop anyway should focus more directly on embodiment.} \simo{I think this is important variable as if we have only one process (generating one artefacts) throughout the study it is harder to say how much the assessment actually looked at the single artefact and derived decisions from that and how much there overall process of the system (generating various artefacts) was considered. By having a small set of different generation processes (each generating a single artefact), we may be able to overcome some arguments against this. Of course, the needed resources (participants/answers/analysis) grow with every added variation dimension.}

\subsubsection{Levels included in the study} The number of levels of perception included in the study is decided by the experiment designer. Selecting particular levels of perception enables comparative analysis with respect to the elements of creativity that they are suitable for. For example, choosing levels L1 and L3 allows comparing how only observing what is drawn relates to observing the physically embodied system producing the artefact. By including L2 as part of the study allows to, then, compare also how virtually and physically embodied systems are perceived. Moreover, some levels are suitable for "stacking", i.e. multiple different L4s can be shown to the participants and the results compared to each other or other levels. \anna{I think, when comparing systems it makes sense for the person doing the study to assess if the systems are on the same level of perceptory evidence or not. I think this could potentially be a strength, if this could be used to assess bias in evaluating two different systems, if their evaluation is important, but cannot be guaranteed to be on the same level. This could potentially help the person planning the study to recognize biases and other issues.}

\subsubsection{Number of levels exposed to a participant} The number of levels of perception an individual study participant is exposed to can be more than one. However, the order of the levels of perception should be randomised for each participant to avoid the príming effect \todo{CITE?}. \christian{Call it \enquote{order effects} and omit citation - should be known well enough, and we don't have space for too many references.} Moreover, if the study includes production of exactly the same artefact for each level of perception, then it should be extra careful in showing more than one level of perception to a single participant. \todo{Add a notion of test fatigue if the number is too high.}

\begin{itemize}
    \item What is drawn: The system should not draw exactly the same thing each time, but the drawings should on some level (what level?) be comparable, e.g. in style/complexity/etc.? Or can the system draw the same thing each time for the same participant?
    \item The order the different systems are shown to the participant: The order supposedly matters and needs to be controlled.
    \item Perception angle: Is the participant allowed to move to observe the physical systems?
    \item Environmental factors: May affect the physical system (e.g. floor material, lighting, temperature, etc.)
    \item 
\end{itemize}

\subsection{Adding Different Modalities}

\begin{itemize}
    \item The experiment setup above only deals with visual observations. However, other modalities could also be added to the general experiment setup adding variation to the information available for the study participant. For example, one could add a brief description of the system (or framing for the particular artefact) that is available for the study participant and then looking how the perception of the system changes depending on if the description is available to the study participant and what level of perception the study participant observes.
    \item \simo{Might not need to be its own section, could be a paragraph in Discussion} \anna{Framing is something I do not consider as a modality per se...}
\end{itemize}

\begin{itemize}
    \item Challenges: How many levels of perception can a single participant be exposed to? If the participant is shown more than one system, does the study render useless if the participant knows that the systems are the same (i.e. how prevalent this kind of study can be)? If only one level of perception can be shown to each participant, how many study participants we need for statistically relevant results?
    \item The experiment setting, as described, relies on the generation process to be (nearly) deterministic so that each participant can be show the same process (or one of the handful pre-selected processes which all produce a distinct artefact) and the same system. This can be achieved in computational systems typically by fixing all random seeds. If this is not achievable or desirable, then it might make more sense to record the generation process for each level of perception to video, and showing the recording to the participants. This, of course, will affect the perception as it could be seen as less engaging.
    \item \christian{Complement determinism of underlying process with fidelity of actuators / sensors.}
    \item \christian{Viewpoints: some types of embodiment afford to be perceived in more varied ways than others, in particular physical embodiment can typically be perceived from different angles while virtual embodiment traditionally is perceived through a flat screen. This can be mitigated by either fixating the participant's position or allowing VR access to the virtual simulation. The risk of the latter is to blend conventional ideas of physical and virtual embodiment - considering virtual embodiment perceived through VR a third embodiment type might be more appropriate.}
\end{itemize}

\end{comment}

\section{Conclusions and Future Work}

We have presented an experimental framework for studying the effects of embodiment and perceptual evidence on the human assessment of creativity. The framework isolates the impact of embodiment on the perception of creativity from its impact on creativity per se by controlling for the artefact production process between conditions. Using the experimental framework, we can answer many interesting questions about the perceived creativity of an embodied system, e.g. \enquote{How does observing the artefact production process affect the creativity assessment when compared to observing only the finished artefact?}, or \enquote{How does more human-like embodiment impact the perceived creativity assessment?}.

The core framework presented in this paper can be further extended. For example, we have focused on visual observations, but allowing other or multi-modal observations is also possible.
Further, other types of evidence, such as framing -- a piece of text covering the motivation behind the artefact and other not directly observable information about the system -- can be treated as a controlled or independent variable.

Our immediate next step is to instantiate this framework and perform a first principled study on the effect of embodiment and perceptual evidence on the human perception of computational creativity. Our focus will be on comparing the effect of different virtual and physical embodiments on the assessment creativity, and on how different kinds of perceptual evidence moderate this effect. Another goal for future work is to use the framework on an existing, established embodied CC system to allow assessing its perceived creativity either with different embodiment modifications or different perceptual evidence. Finally, we would like to test the flexibility of this framework by conducting an experiment outside the lab e.g. in an exhibition space, in order to improve the ecological validity of our findings.

\subsection{Acknowledgments}
SL and AK were funded by the Academy of Finland (AoF, project \#328729), and CG at an early stage by AoF programme “Finnish Center for Artificial Intelligence” (FCAI).


\begin{comment}
\section{Main ideas / notes}
\simo{Some of these are incorporated to the text above, some are not.}
\begin{itemize}
    \item Motivate investigation of virtual vs. physical embodiment (not targeted by workshop organisers) by recent survey: 
    \item Distinguish implicitly from experimental setup at workshop: \enquote{the first day there will be the performances by Daniel and Renaud Chabrier. During the performances, we will ask participants to fill out a questionnaire aimed at testing the general perception of creativity and other dimensions, on the basis of the different modalities of embodiment presented in the performances}.
    \item Potential points to discuss: 
    \begin{itemize}
        \item Setup (what is/are the independent variables - IV=embodiments,…; what are the DVs=perception…)
        \item What is kept static (actual creative output) and why (separation embodiment-creativity vs. embodiment-creativity perception)
        \item What is repeated (creative process, outputs) and why
        \item What’s the protocol, e.g. how is the participant located relative to the experiment: can they walk around it, are they sat in one place, is there head even fixated (well, that’s an extreme scenario…)
        \item What are the technical challenges: measuring perception of creativity; reproducing same output despite noise of physical sensors/actuators or absence of such noise in simulation environments; and how do we aim to address them? 
        \item What are the conceptual challenges: appropriate, unbiased creativity questionnaire; drawing a stimuli that's neither too simplistic (potentially not perceived as creative), nor too complex (to e.g. by drawn by low-fidelity robot without creating differences to output of other, esp. virtually embodied systems).
    \end{itemize}
    \item Use one system as example
    \item Potentially: Include evidence table / characterisation for differences between systems.
    \item Potentially: picture illustrating need to distinguish impact embodiment - creativity vs. embodiment - perception of creativity. 
\end{itemize}
\fi 

\iffalse
\section{HIIT Application}

\simo{Copy-paste}
Computational creativity is a subfield of AI focused on the design and study of AI systems capable of exhibiting creative behavior. Our proposal focuses on one of the field’s most important and most debated questions: how can we evaluate the creativity of an AI system? Our goal is to improve current computational creativity evaluation practices and to support designing better systems for human users by investigating the factors contributing to the human perception of creativity in AI systems.

Our proposal aims to overcome three shortcomings in the present study of computational creativity evaluation: Firstly, most existing evaluation approaches focus on the outputs of computationally creative systems rather than the systems themselves, or the creative processes they engage in. This impedes the meaningful comparison of different systems or different versions of the same system as comparing different systems based on output alone will give only partial information about the creativity of the system, and, thus, does not adequately support making design decisions considering the system. Secondly, while humans have been shown to be biased when evaluating human and machine creativity when presented with human and machine generated output in the same experiment, we know very little about their biases when evaluating different machines only. Finally, no existing work controls for the role of embodiment in evaluation, i.e. how the specific implementation of a specific system interacts with their environment and thus how it is perceived by their evaluator. We hypothesize that differences in embodiment, e.g. a virtual agent and a humanoid robot, have a tremendous impact on how people evaluate the creativity of the respective system.

Our proposed activity consists of a study to investigate whether and how much an AI system’s embodiment affects the human perception of its creativity. The study focuses on the creative process rather than the product, and on the evaluation of AI creativity only, rather than comparing AI creativity to human creativity. More specifically, we aim to compare the human evaluation of three systems: 

1) A non-embodied system only showing a drawing gradually appearing on a screen 

2) A simulated embodied system drawing in a virtual world

3) A physically embodied system drawing on paper

Crucially, all systems implement the same algorithm for drawing to mimic an identical creative process. 
The study will be conducted in a laboratory setting, in which the human evaluators will be presented with the three systems in a randomized order, watching each system complete a drawing. After each  demonstration, they are asked to rate the system’s creativity on a questionnaire derived from state-of-the-art (computational) creativity research. The results are analyzed and compared through exploratory and inferential statistics. Finally each evaluator is asked to compare the systems and interviewed about the differences to elicit rich qualitative feedback on the evaluated systems.

We seek funding for one MSc thesis worker to aid in developing the three systems to be compared. The money will be primarily used for the salary of the MSc thesis worker to enable full time work on the project. We will primarily use the resources of the newly established robotics lab of Software Factory at the University of Helsinki. We however reserve a small portion of the funding for immediate material costs not covered by the existing research infrastructure, such as additional motors and artistic mediums required for the study, small incentives for study participants and refreshments during the experiment. The non-perishable materials will be added to the existing research infrastructure for further use. The implementation process itself will also produce relevant information about the affordances of different embodiments to creativity, while the final human-evaluation may partly be conducted after the original funding period ends.
\end{comment}








\bibliographystyle{iccc}
\small
